Sorting output of du command.

Posted: 01-15-2009, 12:59 PM

Doug Laidlaw

Guest
Posts: n/a

When I use du to list sub-directories (quite a few), the output order is
apparently random. The sub-directories are all named yyyy-mm-dd, but are
not in date or ASCII order. Nor are they in order of size. They may be
adjacent on the disk, but the most recent ones are not first or last. man
du doesn't help. I can't find again what the Linux Cookbook says, but it
was wrong as well. (Although "updated," it still seems out of date. This
isn't the only difference, but it is still good value.)

What is the order? The filesystem is ext3. If defrag is unnecessary, does
the system pick a gap to take the new addition?

TIA,

Doug.
--
I grow old, always learning new things.
-Solon, Greek lawgiver, 6th c. B.C.
And the older I get, the more I need to learn! -DL

Responses to "Sorting output of du command."

Aragorn

Guest
Posts: n/a

Re: Sorting output of du command.

Posted: 01-15-2009, 03:04 PM

On Thursday 15 January 2009 14:59, someone identifying as *Doug Laidlaw*
wrote in /alt.os.linux.mandriva:/

> When I use du to list sub-directories (quite a few), the output order is
> apparently random. The sub-directories are all named yyyy-mm-dd, but are
> not in date or ASCII order. Nor are they in order of size. They may be
> adjacent on the disk, but the most recent ones are not first or last. man
> du doesn't help. I can't find again what the Linux Cookbook says, but it
> was wrong as well. (Although "updated," it still seems out of date. This
> isn't the only difference, but it is still good value.)

I can't really tell you what the order is either, but perhaps it's got to do
with /mtime,/ /ctime/ or /atime?/

> What is the order? The filesystem is ext3.

/ext3/ is an index-sequential filesystem. A balanced tree can be enabled
using /tune2fs/ but is left disabled by default. As such, I would suspect
that the files are listed in the order as they are on disk.

> If defrag is unnecessary, does the system pick a gap to take the new
> addition?

Yes, but in addition, fragmentation is rather irrelevant in a multitasking
and multiuser context, unless the percentage of fragmentation is too high,
of course. The thing is that the kernel will always reschedule disk I/O
operations so that a minimum of head movement is required for a maximum of
throughput. For /ext2/ and /ext3,/ the anticipatory scheduler performs
rather well, but for /reiserfs/ and /XFS/ (and possibly /JFS/) the
completely fair queuing scheduler seems to perform better.

You have to keep in mind that GNU/Linux is multiuser and multitasking, which
means that multiple processes might have concurrent disk I/O going on for
different data blocks, and so fragmentation is less of an issue. As said,
the kernel takes care of the scheduling and it does that quite well.

But yes, as you were asking, the kernel does make sure that fragmentation
occurs as little as possible by the smart allocation of free diskspace. In
balanced tree filesystems, this is done even differently.

--
*Aragorn*
(registered GNU/Linux user #223157)

David W. Hodgins

Guest
Posts: n/a

Re: Sorting output of du command.

Posted: 01-15-2009, 04:09 PM

On Thu, 15 Jan 2009 08:59:42 -0500, Doug Laidlaw <doug@dougshost.invalid> wrote:

> What is the order? The filesystem is ext3. If defrag is unnecessary, does
> the system pick a gap to take the new addition?

The order is the order of the entries in the directory. Same as shown by ls -laU.

Use "du -ak | sort -nr | less" to sort the output in descending size order.

As I understand it, the order of the entries in the directory depends on the
filesystem used, and, in some cases, the options chosen. For ext2/3, the
"tune2fs -E hash_alg xxxx" can be used to set the hash algorithm to legacy,
half_md4, or tea. With legacy, the first available entry will be used. With
a hash used, that will affect the order.

If you really want the details, check out /usr/src/linux/fs/ext3/namei.c.

Regards, Dave Hodgins

--
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

Robert Riches

Guest
Posts: n/a

Re: Sorting output of du command.

Posted: 01-16-2009, 04:33 AM

On 2009-01-15, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote:

> On Thu, 15 Jan 2009 08:59:42 -0500, Doug Laidlaw <doug@dougshost.invalid> wrote:
>
>> What is the order? The filesystem is ext3. If defrag is unnecessary, does
>> the system pick a gap to take the new addition?
>
> The order is the order of the entries in the directory. Same as shown by ls -laU.
>
> Use "du -ak | sort -nr | less" to sort the output in descending size order.
>
> As I understand it, the order of the entries in the directory depends on the
> filesystem used, and, in some cases, the options chosen. For ext2/3, the
> "tune2fs -E hash_alg xxxx" can be used to set the hash algorithm to legacy,
> half_md4, or tea. With legacy, the first available entry will be used. With
> a hash used, that will affect the order.
>
> If you really want the details, check out /usr/src/linux/fs/ext3/namei.c.
>
> Regards, Dave Hodgins

There is a trick I learned nearly 20 years ago (on DEC
Ultrix if I remember correctly) for cases where it is
desirable to force the order of results from du:

1) Rename the top-level directory in question to a different
name. Alternatively, move all the contents somewhere else.

2) Move the child directories into the (new) parent starting
with the one you want to appear first.

3) Check every two or three to make sure the order is what
you want. If not, move the last child directories back out,
do mkdir with a short name to occupy a slot, then move the
child back in.

4) After all children are moved into place, rmdir the
temporary directories you created to occupy slots.

Yes, that method is a pain.

--
Robert Riches
spamtrap42@verizon.net
(Yes, that is one of my email addresses.)

Eric

Guest
Posts: n/a

Re: Sorting output of du command.

Posted: 01-16-2009, 06:38 AM

David W. Hodgins wrote:

> On Thu, 15 Jan 2009 08:59:42 -0500, Doug Laidlaw <doug@dougshost.invalid>
> wrote:
>
>> What is the order? The filesystem is ext3. If defrag is unnecessary,
>> does the system pick a gap to take the new addition?
>
> The order is the order of the entries in the directory. Same as shown by
> ls -laU.
>
> Use "du -ak | sort -nr | less" to sort the output in descending size
> order.
>
> As I understand it, the order of the entries in the directory depends on
> the
> filesystem used, and, in some cases, the options chosen. For ext2/3, the
> "tune2fs -E hash_alg xxxx" can be used to set the hash algorithm to
> legacy,
> half_md4, or tea. With legacy, the first available entry will be used.
> With a hash used, that will affect the order.
>
> If you really want the details, check out /usr/src/linux/fs/ext3/namei.c.
>
> Regards, Dave Hodgins
>

In addition:
you could make an alias (see /etc/profile.d/alias.sh for examples)
alias dus='du -ak | sort -nr'
and put that in your ~/.bash_profile or some such similar place
then whenever you typed dus it would be the same as if you had typed:
du -ak | sort -nr
Eric

LinkBack

Thread Tools

Display Modes

« Previous Thread | Next Thread »

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Posting Rules
You may not post new threads You may post replies You may not post attachments You may not edit your posts BB code is On Smilies are Off [IMG] code is On HTML code is Off Trackbacks are On Pingbacks are On Refbacks are On

Forum Jump

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Command output to file	André Landreau	Windows Vista File Management	3	11-20-2007 09:54 PM
backgrounding a command without console output?	Beowulf.1000	Mandriva Linux	19	10-30-2006 04:22 AM
Output Event Log From Command Line	senrabdet	Windows Vista	4	08-15-2006 02:23 AM
setting an environment variable via the output of a command	Timothy Stone	Windows XP Basics	1	01-13-2004 12:19 PM
initialize a string variable with output of a command?	Glen	Linux / Unix	7	09-29-2003 12:12 PM