gfarm-discuss Mailing List for Gfarm File System

Hi Folks;

I want to build a small HPC with Gfarm FS. All nodes have local hard drives (500GB). What should be the example partition scheme?
Should I have strach directory on local disk or on GfarmFS?

Do you have any comparition document for Gfarm, OrangeFs and LizardFS?

Regards
Taner Karagol


######################################################################
Dikkat:

Bu elektronik posta mesaji kisisel ve ozeldir. Eger size 
gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. 
Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, 
guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki 
gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi 
gorusu olmak zorunda degildir.

######################################################################
Attention: 

This e-mail message is privileged and confidential. If you are 
not the intended recipient please delete the message and notify 
the sender. E-mails to and from the company are monitored for 
operational reasons and in accordance with lawful business practices. 
Any views or opinions presented are solely those of the author and 
do not necessarily represent the views of the company.

######################################################################

Hi Osamu,
Does it mean, that the client will not be able to start reading the
file until the replication is finished? It could be very useful to
change this behaviour such that the file is opened directly from
network immediately, but also the replica is created by dumping the
data which is being
 transferred (in parallel). This way the file will be still
transferred once, the will not be any opening delay caused by the
replication process (which can be long for big files) and when the
file is accessed again by the same client, it will be fast (local).
For example this might be useful for scenarios when a client first
reads first 100MB of a big file, then goes to the first 10MB again,
then reads last 1000MB, then writes first 20MB and so on... Currently
such a scenario works fine (but not very efficient) if performed over
network, but becomes slow if GFARM_FLAGS=r is set. Would be great if
benefits of fast opening (over network) and fast repetitive access
(local) could be combined.
Best regards, Mykola Dimura.


On 1 February 2016 at 02:01, Osamu Tatebe <ta...@cs...> wrote:
> Hi Mykola,
>
> In this case, the client replicates it to the local host,
> and access it.  Currently the client does not access the
> file immediately since replication is more efficient than
> remote access in general, while this behavior can be
> changed easily if it is useful.
>
> Regards,
> Osamu
>
> From: Mykola Dimura <myk...@gm...>
> Subject: Re: [Gfarm-discuss] Adaptive replication
> Date: Sun, 31 Jan 2016 13:12:46 +0100
>
>> Oh, thank you for your reply! According to gfarm_environ "gfarm
>> clients replicates files to local node, before accessing them". Does
>> it mean that the client will not be able to read the file contents
>> before the whole file is copied or it will be accessed immediately
>> from the remote host and the replication will happen in parallel?
>> Best regards, Mykola Dimura.
>>
>>
>> On 31 January 2016 at 09:43, Osamu Tatebe <ta...@cs...> wrote:
>>> Hi Mykola,
>>>
>>> When you specify 'r' by the GFARM_FLAGS environment variable,
>>> the file replica is automatically created to the local host
>>> if the client is executed on the file system node.
>>>
>>> See gfarm_environ(5) for additional information.
>>>
>>> Regards,
>>> Osamu
>>>
>>> From: Mykola Dimura <myk...@gm...>
>>> Subject: [Gfarm-discuss] Adaptive replication
>>> Date: Sat, 30 Jan 2016 13:26:59 +0100
>>>
>>>> Dear Gfarm community,
>>>> In the old Gfarm presentation* by Osamu Tatebe it is mentioned that it
>>>> is possible to use "Adaptive replica creation ... to increase
>>>> performance". Does it mean that Gfarm can dynamically create
>>>> additional replicas to increase performance? For example if there are
>>>> three nodes A, B and C, and a file has two replicas residing at nodes
>>>> A and B and then node C tries to read (or write) the file, is it
>>>> possible to configure Gfarm in a way that an additional replica is
>>>> created automatically at node C for this purpose to increase
>>>> performance?
>>>>
>>>> * http://www.irisa.fr/myriads/alebre/DOWNLOADS-JTE/HPC-FS-Irisa-Gfarm.pdf
>>>> Best regards, Mykola Dimura.
>>
>> ------------------------------------------------------------------------------
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>> _______________________________________________
>> Gfarm-discuss mailing list
>> Gfa...@li...
>> https://lists.sourceforge.net/lists/listinfo/gfarm-discuss

Hi Mykola,

In this case, the client replicates it to the local host,
and access it.  Currently the client does not access the
file immediately since replication is more efficient than
remote access in general, while this behavior can be
changed easily if it is useful.

Regards,
Osamu

From: Mykola Dimura <myk...@gm...>
Subject: Re: [Gfarm-discuss] Adaptive replication
Date: Sun, 31 Jan 2016 13:12:46 +0100

> Oh, thank you for your reply! According to gfarm_environ "gfarm
> clients replicates files to local node, before accessing them". Does
> it mean that the client will not be able to read the file contents
> before the whole file is copied or it will be accessed immediately
> from the remote host and the replication will happen in parallel?
> Best regards, Mykola Dimura.
> 
> 
> On 31 January 2016 at 09:43, Osamu Tatebe <ta...@cs...> wrote:
>> Hi Mykola,
>>
>> When you specify 'r' by the GFARM_FLAGS environment variable,
>> the file replica is automatically created to the local host
>> if the client is executed on the file system node.
>>
>> See gfarm_environ(5) for additional information.
>>
>> Regards,
>> Osamu
>>
>> From: Mykola Dimura <myk...@gm...>
>> Subject: [Gfarm-discuss] Adaptive replication
>> Date: Sat, 30 Jan 2016 13:26:59 +0100
>>
>>> Dear Gfarm community,
>>> In the old Gfarm presentation* by Osamu Tatebe it is mentioned that it
>>> is possible to use "Adaptive replica creation ... to increase
>>> performance". Does it mean that Gfarm can dynamically create
>>> additional replicas to increase performance? For example if there are
>>> three nodes A, B and C, and a file has two replicas residing at nodes
>>> A and B and then node C tries to read (or write) the file, is it
>>> possible to configure Gfarm in a way that an additional replica is
>>> created automatically at node C for this purpose to increase
>>> performance?
>>>
>>> * http://www.irisa.fr/myriads/alebre/DOWNLOADS-JTE/HPC-FS-Irisa-Gfarm.pdf
>>> Best regards, Mykola Dimura.
> 
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> _______________________________________________
> Gfarm-discuss mailing list
> Gfa...@li...
> https://lists.sourceforge.net/lists/listinfo/gfarm-discuss

Oh, thank you for your reply! According to gfarm_environ "gfarm
clients replicates files to local node, before accessing them". Does
it mean that the client will not be able to read the file contents
before the whole file is copied or it will be accessed immediately
from the remote host and the replication will happen in parallel?
Best regards, Mykola Dimura.


On 31 January 2016 at 09:43, Osamu Tatebe <ta...@cs...> wrote:
> Hi Mykola,
>
> When you specify 'r' by the GFARM_FLAGS environment variable,
> the file replica is automatically created to the local host
> if the client is executed on the file system node.
>
> See gfarm_environ(5) for additional information.
>
> Regards,
> Osamu
>
> From: Mykola Dimura <myk...@gm...>
> Subject: [Gfarm-discuss] Adaptive replication
> Date: Sat, 30 Jan 2016 13:26:59 +0100
>
>> Dear Gfarm community,
>> In the old Gfarm presentation* by Osamu Tatebe it is mentioned that it
>> is possible to use "Adaptive replica creation ... to increase
>> performance". Does it mean that Gfarm can dynamically create
>> additional replicas to increase performance? For example if there are
>> three nodes A, B and C, and a file has two replicas residing at nodes
>> A and B and then node C tries to read (or write) the file, is it
>> possible to configure Gfarm in a way that an additional replica is
>> created automatically at node C for this purpose to increase
>> performance?
>>
>> * http://www.irisa.fr/myriads/alebre/DOWNLOADS-JTE/HPC-FS-Irisa-Gfarm.pdf
>> Best regards, Mykola Dimura.

Hi Mykola,

When you specify 'r' by the GFARM_FLAGS environment variable,
the file replica is automatically created to the local host
if the client is executed on the file system node.

See gfarm_environ(5) for additional information.

Regards,
Osamu

From: Mykola Dimura <myk...@gm...>
Subject: [Gfarm-discuss] Adaptive replication
Date: Sat, 30 Jan 2016 13:26:59 +0100

> Dear Gfarm community,
> In the old Gfarm presentation* by Osamu Tatebe it is mentioned that it
> is possible to use "Adaptive replica creation ... to increase
> performance". Does it mean that Gfarm can dynamically create
> additional replicas to increase performance? For example if there are
> three nodes A, B and C, and a file has two replicas residing at nodes
> A and B and then node C tries to read (or write) the file, is it
> possible to configure Gfarm in a way that an additional replica is
> created automatically at node C for this purpose to increase
> performance?
> 
> * http://www.irisa.fr/myriads/alebre/DOWNLOADS-JTE/HPC-FS-Irisa-Gfarm.pdf
> Best regards, Mykola Dimura.

Dear Gfarm community,
In the old Gfarm presentation* by Osamu Tatebe it is mentioned that it
is possible to use "Adaptive replica creation ... to increase
performance". Does it mean that Gfarm can dynamically create
additional replicas to increase performance? For example if there are
three nodes A, B and C, and a file has two replicas residing at nodes
A and B and then node C tries to read (or write) the file, is it
possible to configure Gfarm in a way that an additional replica is
created automatically at node C for this purpose to increase
performance?

* http://www.irisa.fr/myriads/alebre/DOWNLOADS-JTE/HPC-FS-Irisa-Gfarm.pdf
Best regards, Mykola Dimura.

Hi Zhang,

It seems your Gfarm is working.

> [mca@clu27 bin]$ gfreg /home/mca/27file test
> [mca@clu28 bin]$ gfls -l
> -rw-r--r-- 1 mca      gfarmadm          6 Jan 11 22:28 test

This means the local file /home/mca/27file is registered
(or copied) to 'test' (exactly '/test') in Gfarm file system.

You can copy it from the Gfarm file system by

% gfexport test > localfile

or if you have many files,

% gfpcopy gfarm:///test localdir

For details, refer to the manual page of gfpcopy.

> [mca@clu28 bin]$ gfwhere test
> clu27

This means the file 'test' is stored at clu27.

BTW, if you use gfarm2fs, you can mount Gfarm.  In this
case you can access files in Gfarm file system by usual
commands.

Regards,
Osamu

From: "zjm...@16..." <zjm...@16...>
Subject: [Gfarm-discuss] Can I use Gfarm to achieve distributed file operations ?
Date: Mon, 11 Jan 2016 22:14:22 +0800

> Hi，
> I am just a green hand for Gfarm,   I want to achieve remote file  operation
> by using it, such as remote download or remote delete. Can I?
>  
> I test it on two node(clu27 & clu28). I start gfmd and gfsd on clu28 and start
> gfsd on clu27.
> [mca@clu28 bin]$ gfhost -l
> 0.42/0.40/0.36 s x86_64-centos6.5-linux 16 clu27 600 0(10.3.10.27)
> 0.47/0.41/0.37 s x86_64-centos6.5-linux 16 clu28 600 0(10.3.10.28)
> 
> I want to achieve remote file copy/download to local, but failed. 
> 
> I gfreg a local file named test from node clu27,I can cat it in clu28. but how
> can I downlaod it to clu28 local system by use Gfarm?
> [mca@clu27 bin]$ gfreg /home/mca/27file test
> [mca@clu28 bin]$ gfls -l
> -rw-r--r-- 1 mca      gfarmadm          6 Jan 11 22:28 test
> 
> Can I remove the file in clu27:/home/mca/27file by using Gfarm client on
> clu28, The following seems ineffective
> [mca@clu28 bin]$ gfrm -f test
> [mca@clu27 bin]$ gfls
> [mca@clu27 bin]$ ll /home/mca/27file 
> -rw-r--r-- 1 mca ghips 6 Jan 11 21:51 /home/mca/27file
> 
> In addition, if I want to locate file's absolute path by using Gfarm, but it
> does not work  when I using gwhere, The meta info about /home/mca/27file is
> missing? I cannot find it in PSQL backend DB.
> [mca@clu28 bin]$ gfwhere test
> clu27
> 
> I must be going wrong, Could you help me?
> Looking forward to your reply. Thank you very much.
> Best regards!
>  
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Zhang Jie

Hi，
I am just a green hand for Gfarm,   I want to achieve remote file  operation by using it, such as remote download or remote delete. Can I?
 
I test it on two node(clu27 & clu28). I start gfmd and gfsd on clu28 and start gfsd on clu27.
[mca@clu28 bin]$ gfhost -l
0.42/0.40/0.36 s x86_64-centos6.5-linux 16 clu27 600 0(10.3.10.27)
0.47/0.41/0.37 s x86_64-centos6.5-linux 16 clu28 600 0(10.3.10.28)

I want to achieve remote file copy/download to local, but failed. 

I gfreg a local file named test from node clu27,I can cat it in clu28. but how can I downlaod it to clu28 local system by use Gfarm?
[mca@clu27 bin]$ gfreg /home/mca/27file test
[mca@clu28 bin]$ gfls -l
-rw-r--r-- 1 mca      gfarmadm          6 Jan 11 22:28 test

Can I remove the file in clu27:/home/mca/27file by using Gfarm client on clu28, The following seems ineffective
[mca@clu28 bin]$ gfrm -f test
[mca@clu27 bin]$ gfls
[mca@clu27 bin]$ ll /home/mca/27file 
-rw-r--r-- 1 mca ghips 6 Jan 11 21:51 /home/mca/27file

In addition, if I want to locate file's absolute path by using Gfarm, but it does not work  when I using gwhere, The meta info about /home/mca/27file is missing? I cannot find it in PSQL backend DB.
[mca@clu28 bin]$ gfwhere test
clu27

I must be going wrong, Could you help me?
Looking forward to your reply. Thank you very much.
Best regards!
 


Zhang Jie

	Hi Osamu,

Thank you very much for these clarifications.

Best regards,

-- 
Olivier Cherrier - Symacx.com
mailto:oc...@sy...

On Fri, Dec 11, 2015 at 05:54:44PM +0900, ta...@cs... wrote:
> 
> Hi Olivier,
> 
> We are assuming the close-to-open consistency in Gfarm
> like AFS.  That means every replica needs not to be
> updated 'live' before closing a file.  We take this
> advantage to improve the performance.
> 
> Our motivation of the Gfarm file system is to process
> files locally as long as possible.  Most applications
> except mapreduce and MPI-IO apps, read input files,
> and write some output files.  Pwrake workflow system
> automatically selects nodes to minimize the data
> transfer among nodes considering locations of input
> files in a whole workflow.  When spliting files into
> chunks, remote accesses may be required to access them
> that may impact the performance.
> 
> BTW, Gfarm hadoop plugin and Gfarm MPI-IO plugin are
> available for mapreduce and MPI-IO applications.
> 
> Regards,
> Osamu
> 
> From: Olivier Cherrier <oc...@sy...>
> Subject: [Gfarm-discuss] write operations on replicas
> Date: Wed, 9 Dec 2015 15:34:27 +0100
> 
> > 
> > 	Hi,
> > 
> > I am currently reviewing / testing Gfarm and I would have a question
> > regarding this section of 'SETUP.en':
> > 
> > ----8<----
> > File replicas will be created after the close operation when the file
> > is created, or updated.
> > ----8<----
> > 
> > Does it mean replicas are only updated when the file is closed?
> > Or the replicas are updated 'live' while a node is writing to the file?
> > 
> > 
> > BTW, is it possible to split big files accorss multiple file system
> > nodes ?
> > 
> > 
> > Thank you very much,
> > Best regards
> > 
> > -- 
> > Olivier Cherrier - Symacx.com
> > mailto:oc...@sy...
> > 
> > ------------------------------------------------------------------------------
> > _______________________________________________
> > Gfarm-discuss mailing list
> > Gfa...@li...
> > https://lists.sourceforge.net/lists/listinfo/gfarm-discuss

Hi Olivier,

We are assuming the close-to-open consistency in Gfarm
like AFS.  That means every replica needs not to be
updated 'live' before closing a file.  We take this
advantage to improve the performance.

Our motivation of the Gfarm file system is to process
files locally as long as possible.  Most applications
except mapreduce and MPI-IO apps, read input files,
and write some output files.  Pwrake workflow system
automatically selects nodes to minimize the data
transfer among nodes considering locations of input
files in a whole workflow.  When spliting files into
chunks, remote accesses may be required to access them
that may impact the performance.

BTW, Gfarm hadoop plugin and Gfarm MPI-IO plugin are
available for mapreduce and MPI-IO applications.

Regards,
Osamu

From: Olivier Cherrier <oc...@sy...>
Subject: [Gfarm-discuss] write operations on replicas
Date: Wed, 9 Dec 2015 15:34:27 +0100

> 
> 	Hi,
> 
> I am currently reviewing / testing Gfarm and I would have a question
> regarding this section of 'SETUP.en':
> 
> ----8<----
> File replicas will be created after the close operation when the file
> is created, or updated.
> ----8<----
> 
> Does it mean replicas are only updated when the file is closed?
> Or the replicas are updated 'live' while a node is writing to the file?
> 
> 
> BTW, is it possible to split big files accorss multiple file system
> nodes ?
> 
> 
> Thank you very much,
> Best regards
> 
> -- 
> Olivier Cherrier - Symacx.com
> mailto:oc...@sy...
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> Gfarm-discuss mailing list
> Gfa...@li...
> https://lists.sourceforge.net/lists/listinfo/gfarm-discuss

	Hi,

I am currently reviewing / testing Gfarm and I would have a question
regarding this section of 'SETUP.en':

----8<----
File replicas will be created after the close operation when the file
is created, or updated.
----8<----

Does it mean replicas are only updated when the file is closed?
Or the replicas are updated 'live' while a node is writing to the file?


BTW, is it possible to split big files accorss multiple file system
nodes ?


Thank you very much,
Best regards

-- 
Olivier Cherrier - Symacx.com
mailto:oc...@sy...