Discussion:
[uClinux-dev] ls: can't open '.': Cannot allocate memory
Luis Alves
2012-10-29 15:32:34 UTC
Permalink
Hi,

Just like the mail subject says, I'm getting a lot of these errors
when 'ls'ing' NFS mounts (but only NFS mounts).

Full error sample text:

# ls
ls: page allocation failure: order:8, mode:0xd0
Call Trace:
[<008446e8>] 0x8446e8
[<0084683a>] 0x84683a
[<00851e34>] 0x851e34
[<008526c2>] 0x8526c2
[<008526c2>] 0x8526c2

[<0084e200>] 0x84e200
[<0085270a>] 0x85270a
[<008127d2>] 0x8127d2

Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:0 inactive_anon:0 isolated_anon:0
active_file:0 inactive_file:16 isolated_file:0
unevictable:1 dirty:0 writeback:0 unstable:0
free:1340 slab_reclaimable:24 slab_unreclaimable:255
mapped:0 shmem:0 pagetables:0 bounce:0
free_cma:0
DMA free:5360kB min:500kB low:624kB high:748kB active_anon:0kB
inactive_anon:0kB active_file:0kB inactive_file:64kB unevictable:4kB
isolated(anon):0kB isolated(file):0kB present:15704kB mlocked:0kB
dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:96kB
slab_unreclaimable:1020kB kernel_stack:84kB pagetables:0kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
17 total pagecache pages
2046 pages RAM
152 pages reserved
261998 pages shared
541 pages non-shared
Allocation of length 528384 from process 94 (ls) failed
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:0 inactive_anon:0 isolated_anon:0
active_file:0 inactive_file:16 isolated_file:0
unevictable:1 dirty:0 writeback:0 unstable:0
free:1340 slab_reclaimable:24 slab_unreclaimable:255
mapped:0 shmem:0 pagetables:0 bounce:0
free_cma:0
DMA free:5360kB min:500kB low:624kB high:748kB active_anon:0kB
inactive_anon:0kB active_file:0kB inactive_file:64kB unevictable:4kB
isolated(anon):0kB isolated(file):0kB present:15704kB mlocked:0kB
dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:96kB
slab_unreclaimable:1020kB kernel_stack:84kB pagetables:0kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
17 total pagecache pages
ls: can't open '.': Cannot allocate memory
#


call trace address description:

[<008446e8>] 0x8446e8 (inside <warn_alloc_failed>)
[<0084683a>] 0x84683a (inside <__alloc_pages_nodemask>)
[<00851e34>] 0x851e34 (inside <do_mmap_pgoff>)
[<008526c2>] 0x8526c2 <sys_old_mmap>
[<008526c2>] 0x8526c2 <sys_old_mmap>

[<0084e200>] 0x84e200 (inside <vm_mmap_pgoff>)
[<0085270a>] 0x85270a (inside <sys_old_mmap>)
[<008127d2>] 0x8127d2 (inside <enosys>)


Usually I have around 4 ~ 5 Mb free memory, so even if 'ls' needs
around 500k (like it says it needs in the dump - Allocation of length
528384 from process 94 (ls) failed) it would be enough.
Using busybox 'ls'.

Any sugestion?

Regards,
Luis
Larry Baker
2012-10-29 17:49:27 UTC
Permalink
Luis,
Post by Luis Alves
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
The largest block of memory your system has available is 524288 (512*1024).
Post by Luis Alves
Allocation of length 528384 from process 94 (ls) failed
The requested size is larger than that, so uClinux tries to allocate the next larger block size (1024K). There are none available.
Post by Luis Alves
Any sugestion?
I do not know what determines how much memory ls allocates.

I have similar memory allocation failures on my system. It does not appear to use a shared uclibc. I do not know (yet) how to enable building a shared uclibc or how to change the builds to use it. If the instructions are straightforward, I would like to hear from someone.
Post by Luis Alves
Regards,
Luis
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.uclinux.org/pipermail/uclinux-dev/attachments/20121029/46f73e0c/attachment.html>
Luis Alves
2012-10-29 19:38:28 UTC
Permalink
Thanks for the explanation Larry.

Meanwhile I've read some stuff about non-MMU memory alocation (I had
the wrong idea of how memory was allocated).
I guess my system ram is not that much (only 8Mb)...

How would shared libs solve the problem?
The problem is allocating memory for 'ls' data and not 'ls' itself. Is
this correct?


Thanks
Post by Luis Alves
Luis,
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
The largest block of memory your system has available is 524288 (512*1024).
Allocation of length 528384 from process 94 (ls) failed
The requested size is larger than that, so uClinux tries to allocate the
next larger block size (1024K). There are none available.
Any sugestion?
I do not know what determines how much memory ls allocates.
I have similar memory allocation failures on my system. It does not appear
to use a shared uclibc. I do not know (yet) how to enable building a shared
uclibc or how to change the builds to use it. If the instructions are
straightforward, I would like to hear from someone.
Regards,
Luis
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov
Larry Baker
2012-10-29 19:46:28 UTC
Permalink
Luis,
Post by Luis Alves
Thanks for the explanation Larry.
Meanwhile I've read some stuff about non-MMU memory alocation (I had
the wrong idea of how memory was allocated).
I guess my system ram is not that much (only 8Mb)...
I have the same situation. I have a Lantronix EDS2100 which uses a ColdFire 5208 with 8 MB RAM and 8 MB flash.

I was able to obtain more RAM by using the XIP option (enabled by a boot loader command) and by eliminating the RAM tmpfs (I had to edit a uClinux startup file). I do not know if these techniques are possible on your hardware.
Post by Luis Alves
How would shared libs solve the problem?
All programs would share a single copy of uclibc. Presumably, your applications need it, so it would already be loaded when you run the ls command. I have no idea yet how to do this.
Post by Luis Alves
The problem is allocating memory for 'ls' data and not 'ls' itself. Is
this correct?
It looks to me like it is coming from the ls command itself, not from the loader.
Post by Luis Alves
Thanks
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.uclinux.org/pipermail/uclinux-dev/attachments/20121029/98b0fa26/attachment.html>
Greg Ungerer
2012-10-30 00:33:33 UTC
Permalink
Hi Luis,
Post by Luis Alves
Thanks for the explanation Larry.
Meanwhile I've read some stuff about non-MMU memory alocation (I had
the wrong idea of how memory was allocated).
I guess my system ram is not that much (only 8Mb)...
What you are seeing though is a classic case of memory fragmentation.
You have eneough free RAM in total, but not enough in s ingle
contiguous block. On a paged MMU system this is not a problem, pages
can be mapped so you see a contiguous block the size you want. You
can't do that without no MMU and paging.
Post by Luis Alves
How would shared libs solve the problem?
The problem is allocating memory for 'ls' data and not 'ls' itself. Is
this correct?
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.

Regards
Greg
Post by Luis Alves
Post by Luis Alves
Luis,
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
The largest block of memory your system has available is 524288 (512*1024).
Allocation of length 528384 from process 94 (ls) failed
The requested size is larger than that, so uClinux tries to allocate the
next larger block size (1024K). There are none available.
Any sugestion?
I do not know what determines how much memory ls allocates.
I have similar memory allocation failures on my system. It does not appear
to use a shared uclibc. I do not know (yet) how to enable building a shared
uclibc or how to change the builds to use it. If the instructions are
straightforward, I would like to hear from someone.
Regards,
Luis
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close FAX: +61 7 3217 5323
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
Luis Alves
2012-10-30 10:38:52 UTC
Permalink
Hi Greg,
Post by Greg Ungerer
What you are seeing though is a classic case of memory fragmentation.
Thats true. If I spam a lot of 'ls' with other commands between, it
eventually works. (I guess at some point a memory chunk of 1024k
becomes available).
I've tried to rebuild the kernel using as memory allocator SLOB and
SLUB, but the result seems to be the same as using SLAB.
Post by Greg Ungerer
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
I've tried with standalone 'ls' and the result is the same.
It also fails wanting to allocate exacly 528384 (always the same value
and only when ls'ing NFS mounts).
(the mem alloc value is actually pretty rounded 512k + 4k)

Could it be related to NFS kernel code?

Don't know if it's relevant, but I'm using latest kernel sources from
your git, gcc-4.7.2, running kernel from flash (XIP) and user binaries
compiled with -msep-data (running from flash also).


By the way, like Larry pointed out, is there a way to compile user
binaries with shared flat libraries?
Long time ago I've tried to enable UCLIBC_FORMAT_SHARED_FLAT in uClibc
config, but then it was failing building (don't remember where/why it
fails).


Thanks,
Luis
Post by Greg Ungerer
Hi Luis,
Post by Luis Alves
Thanks for the explanation Larry.
Meanwhile I've read some stuff about non-MMU memory alocation (I had
the wrong idea of how memory was allocated).
I guess my system ram is not that much (only 8Mb)...
What you are seeing though is a classic case of memory fragmentation.
You have eneough free RAM in total, but not enough in s ingle
contiguous block. On a paged MMU system this is not a problem, pages
can be mapped so you see a contiguous block the size you want. You
can't do that without no MMU and paging.
Post by Luis Alves
How would shared libs solve the problem?
The problem is allocating memory for 'ls' data and not 'ls' itself. Is
this correct?
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
Regards
Greg
Post by Luis Alves
Post by Luis Alves
Luis,
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
The largest block of memory your system has available is 524288 (512*1024).
Allocation of length 528384 from process 94 (ls) failed
The requested size is larger than that, so uClinux tries to allocate the
next larger block size (1024K). There are none available.
Any sugestion?
I do not know what determines how much memory ls allocates.
I have similar memory allocation failures on my system. It does not appear
to use a shared uclibc. I do not know (yet) how to enable building a shared
uclibc or how to change the builds to use it. If the instructions are
straightforward, I would like to hear from someone.
Regards,
Luis
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close FAX: +61 7 3217 5323
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
Greg Ungerer
2012-10-30 12:03:13 UTC
Permalink
Hi Luis,
Post by Luis Alves
Post by Greg Ungerer
What you are seeing though is a classic case of memory fragmentation.
Thats true. If I spam a lot of 'ls' with other commands between, it
eventually works. (I guess at some point a memory chunk of 1024k
becomes available).
I've tried to rebuild the kernel using as memory allocator SLOB and
SLUB, but the result seems to be the same as using SLAB.
Post by Greg Ungerer
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
I've tried with standalone 'ls' and the result is the same.
It also fails wanting to allocate exacly 528384 (always the same value
and only when ls'ing NFS mounts).
(the mem alloc value is actually pretty rounded 512k + 4k)
Could it be related to NFS kernel code?
It could be. From the stack back trace:

[<008446e8>] 0x8446e8 (inside <warn_alloc_failed>)
[<0084683a>] 0x84683a (inside <__alloc_pages_nodemask>)
[<00851e34>] 0x851e34 (inside <do_mmap_pgoff>)
[<008526c2>] 0x8526c2 <sys_old_mmap>
[<008526c2>] 0x8526c2 <sys_old_mmap>
[<0084e200>] 0x84e200 (inside <vm_mmap_pgoff>)
[<0085270a>] 0x85270a (inside <sys_old_mmap>)

That looks like an mmap system call. And mmap is the underlying
allocator used by malloc() in uClibc (for non-MMU). So to me it
looks like a malloc somewhere in busybox. That is just a first
guess, it needs some further investigation.
Post by Luis Alves
Don't know if it's relevant, but I'm using latest kernel sources from
your git, gcc-4.7.2, running kernel from flash (XIP) and user binaries
compiled with -msep-data (running from flash also).
I wouldn't suspect anything different. But if you can try older kernel
versions you could confirm that.
Post by Luis Alves
By the way, like Larry pointed out, is there a way to compile user
binaries with shared flat libraries?
Long time ago I've tried to enable UCLIBC_FORMAT_SHARED_FLAT in uClibc
config, but then it was failing building (don't remember where/why it
fails).
I haven't used shared libraries in these setups for years. So I am
not sure what state they are in. Seems like it might have bit rotted
from your experiences.

Regards
Greg
Post by Luis Alves
Post by Greg Ungerer
Hi Luis,
Post by Luis Alves
Thanks for the explanation Larry.
Meanwhile I've read some stuff about non-MMU memory alocation (I had
the wrong idea of how memory was allocated).
I guess my system ram is not that much (only 8Mb)...
What you are seeing though is a classic case of memory fragmentation.
You have eneough free RAM in total, but not enough in s ingle
contiguous block. On a paged MMU system this is not a problem, pages
can be mapped so you see a contiguous block the size you want. You
can't do that without no MMU and paging.
Post by Luis Alves
How would shared libs solve the problem?
The problem is allocating memory for 'ls' data and not 'ls' itself. Is
this correct?
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
Regards
Greg
Post by Luis Alves
Post by Luis Alves
Luis,
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
The largest block of memory your system has available is 524288 (512*1024).
Allocation of length 528384 from process 94 (ls) failed
The requested size is larger than that, so uClinux tries to allocate the
next larger block size (1024K). There are none available.
Any sugestion?
I do not know what determines how much memory ls allocates.
I have similar memory allocation failures on my system. It does not appear
to use a shared uclibc. I do not know (yet) how to enable building a shared
uclibc or how to change the builds to use it. If the instructions are
straightforward, I would like to hear from someone.
Regards,
Luis
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close FAX: +61 7 3217 5323
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close, FAX: +61 7 3891 3630
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
Larry Baker
2012-10-30 17:39:38 UTC
Permalink
Greg,
Post by Greg Ungerer
Hi Luis,
Post by Luis Alves
Post by Greg Ungerer
What you are seeing though is a classic case of memory fragmentation.
Thats true. If I spam a lot of 'ls' with other commands between, it
eventually works. (I guess at some point a memory chunk of 1024k
becomes available).
I've tried to rebuild the kernel using as memory allocator SLOB and
SLUB, but the result seems to be the same as using SLAB.
I've tried those solutions too -- no good. I decided that was because those are kernel memory allocation policies, and have no effect on user memory allocation. Is that correct?
Post by Greg Ungerer
Post by Luis Alves
Post by Greg Ungerer
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
I've tried with standalone 'ls' and the result is the same.
It also fails wanting to allocate exacly 528384 (always the same value
and only when ls'ing NFS mounts).
(the mem alloc value is actually pretty rounded 512k + 4k)
Could it be related to NFS kernel code?
[<008446e8>] 0x8446e8 (inside <warn_alloc_failed>)
[<0084683a>] 0x84683a (inside <__alloc_pages_nodemask>)
[<00851e34>] 0x851e34 (inside <do_mmap_pgoff>)
[<008526c2>] 0x8526c2 <sys_old_mmap>
[<008526c2>] 0x8526c2 <sys_old_mmap>
[<0084e200>] 0x84e200 (inside <vm_mmap_pgoff>)
[<0085270a>] 0x85270a (inside <sys_old_mmap>)
That looks like an mmap system call. And mmap is the underlying
allocator used by malloc() in uClibc (for non-MMU). So to me it
looks like a malloc somewhere in busybox. That is just a first
guess, it needs some further investigation.
Post by Luis Alves
Don't know if it's relevant, but I'm using latest kernel sources from
your git, gcc-4.7.2, running kernel from flash (XIP) and user binaries
compiled with -msep-data (running from flash also).
I wouldn't suspect anything different. But if you can try older kernel
versions you could confirm that.
Post by Luis Alves
By the way, like Larry pointed out, is there a way to compile user
binaries with shared flat libraries?
Long time ago I've tried to enable UCLIBC_FORMAT_SHARED_FLAT in uClibc
config, but then it was failing building (don't remember where/why it
fails).
I haven't used shared libraries in these setups for years. So I am
not sure what state they are in. Seems like it might have bit rotted
from your experiences.
Rats.

It seems to me the root of the problem on memory constrained systems is the power-of-2 memory allocator; a boxcar memory allocator would work better, I think. I have read there used to be a non-power-of-2 memory allocator, named page_alloc2. Is there any chance it can be revived for 2.6 kernels? (See my post of 23 October 2012.)
Post by Greg Ungerer
Regards
Greg
Post by Luis Alves
Post by Greg Ungerer
Hi Luis,
Post by Luis Alves
Thanks for the explanation Larry.
Meanwhile I've read some stuff about non-MMU memory alocation (I had
the wrong idea of how memory was allocated).
I guess my system ram is not that much (only 8Mb)...
What you are seeing though is a classic case of memory fragmentation.
You have eneough free RAM in total, but not enough in s ingle
contiguous block. On a paged MMU system this is not a problem, pages
can be mapped so you see a contiguous block the size you want. You
can't do that without no MMU and paging.
Post by Luis Alves
How would shared libs solve the problem?
The problem is allocating memory for 'ls' data and not 'ls' itself. Is
this correct?
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
Regards
Greg
Post by Luis Alves
Post by Luis Alves
Luis,
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
The largest block of memory your system has available is 524288 (512*1024).
Allocation of length 528384 from process 94 (ls) failed
The requested size is larger than that, so uClinux tries to allocate the
next larger block size (1024K). There are none available.
Any sugestion?
I do not know what determines how much memory ls allocates.
I have similar memory allocation failures on my system. It does not appear
to use a shared uclibc. I do not know (yet) how to enable building a shared
uclibc or how to change the builds to use it. If the instructions are
straightforward, I would like to hear from someone.
Regards,
Luis
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close FAX: +61 7 3217 5323
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close, FAX: +61 7 3891 3630
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.uclinux.org/pipermail/uclinux-dev/attachments/20121030/a596ca68/attachment.html>
Gavin Lambert
2012-10-30 22:04:00 UTC
Permalink
I've tried those solutions too -- no good. ?I decided that was
because those are kernel memory allocation policies, and have no
effect on user memory allocation. ?Is that correct?
Not entirely. User-mode malloc typically requests a chunk of memory from
the kernel (which is subject to kernel allocation policies) and then
subdivides this internally for further requests (from the same process
only). This does mean that with each new process started a bit of memory
becomes unavailable to other processes until they exit; depending on the
pattern of new processes starting and how much memory they're allocating
this can lead to fragmentation.

The precise details vary a bit depending on which libc you're using and how
you have it configured.
Larry Baker
2012-10-30 22:28:17 UTC
Permalink
Post by Gavin Lambert
Post by Larry Baker
I've tried those solutions too -- no good. I decided that was
because those are kernel memory allocation policies, and have no
effect on user memory allocation. Is that correct?
Not entirely. User-mode malloc typically requests a chunk of memory from
the kernel (which is subject to kernel allocation policies) and then
subdivides this internally for further requests (from the same process
only).
I thought I've read somewhere though that SLUB/SLAB/SLOB (the different kernel allocation policies) come into play only when the allocations are small enough (your reference to "subdivides"?) to be made from the minimum chunk actually allocated from RAM allocator. And, that it is the power-of-2 RAM allocator that is causing the problems for memory constrained systems. It is also my impression that SLUB/SLAB/SLOB have no effect on the power-of-2 RAM allocator, or whatever might be used in place of it, e.g., the non-power-of-2 allocator, page_alloc2.
Post by Gavin Lambert
This does mean that with each new process started a bit of memory
becomes unavailable to other processes until they exit; depending on the
pattern of new processes starting and how much memory they're allocating
this can lead to fragmentation.
The precise details vary a bit depending on which libc you're using and how
you have it configured.
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org
http://mailman.uclinux.org/mailman/options/uclinux-dev
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov
Greg Ungerer
2012-10-31 03:11:50 UTC
Permalink
Hi Larry,
Post by Larry Baker
Post by Luis Alves
Post by Luis Alves
Post by Greg Ungerer
What you are seeing though is a classic case of memory fragmentation.
Thats true. If I spam a lot of 'ls' with other commands between, it
eventually works. (I guess at some point a memory chunk of 1024k
becomes available).
I've tried to rebuild the kernel using as memory allocator SLOB and
SLUB, but the result seems to be the same as using SLAB.
I've tried those solutions too -- no good. I decided that was because those are kernel memory allocation policies, and have no effect on user memory allocation. Is that correct?
Post by Luis Alves
Post by Luis Alves
Post by Greg Ungerer
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
I've tried with standalone 'ls' and the result is the same.
It also fails wanting to allocate exacly 528384 (always the same value
and only when ls'ing NFS mounts).
(the mem alloc value is actually pretty rounded 512k + 4k)
Could it be related to NFS kernel code?
[<008446e8>] 0x8446e8 (inside <warn_alloc_failed>)
[<0084683a>] 0x84683a (inside <__alloc_pages_nodemask>)
[<00851e34>] 0x851e34 (inside <do_mmap_pgoff>)
[<008526c2>] 0x8526c2 <sys_old_mmap>
[<008526c2>] 0x8526c2 <sys_old_mmap>
[<0084e200>] 0x84e200 (inside <vm_mmap_pgoff>)
[<0085270a>] 0x85270a (inside <sys_old_mmap>)
That looks like an mmap system call. And mmap is the underlying
allocator used by malloc() in uClibc (for non-MMU). So to me it
looks like a malloc somewhere in busybox. That is just a first
guess, it needs some further investigation.
Post by Luis Alves
Don't know if it's relevant, but I'm using latest kernel sources from
your git, gcc-4.7.2, running kernel from flash (XIP) and user binaries
compiled with -msep-data (running from flash also).
I wouldn't suspect anything different. But if you can try older kernel
versions you could confirm that.
Post by Luis Alves
By the way, like Larry pointed out, is there a way to compile user
binaries with shared flat libraries?
Long time ago I've tried to enable UCLIBC_FORMAT_SHARED_FLAT in uClibc
config, but then it was failing building (don't remember where/why it
fails).
I haven't used shared libraries in these setups for years. So I am
not sure what state they are in. Seems like it might have bit rotted
from your experiences.
Rats.
It seems to me the root of the problem on memory constrained systems is the power-of-2 memory allocator; a boxcar memory allocator would work better, I think. I have read there used to be a non-power-of-2 memory allocator, named page_alloc2. Is there any chance it can be revived for 2.6 kernels? (See my post of 23 October 2012.)
I am sure it could be. But it would take a little effort though.
I think Phil Wilshire made an effort a few years back to bring the
page_alloc2 code up to date with 2.6 kernels. Google and check the
archives you may be able to find it.

Regards
Greg
Post by Larry Baker
Post by Luis Alves
Post by Luis Alves
Post by Greg Ungerer
Hi Luis,
Post by Luis Alves
Thanks for the explanation Larry.
Meanwhile I've read some stuff about non-MMU memory alocation (I had
the wrong idea of how memory was allocated).
I guess my system ram is not that much (only 8Mb)...
What you are seeing though is a classic case of memory fragmentation.
You have eneough free RAM in total, but not enough in s ingle
contiguous block. On a paged MMU system this is not a problem, pages
can be mapped so you see a contiguous block the size you want. You
can't do that without no MMU and paging.
Post by Luis Alves
How would shared libs solve the problem?
The problem is allocating memory for 'ls' data and not 'ls' itself. Is
this correct?
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
Regards
Greg
Post by Luis Alves
Post by Luis Alves
Luis,
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
The largest block of memory your system has available is 524288 (512*1024).
Allocation of length 528384 from process 94 (ls) failed
The requested size is larger than that, so uClinux tries to allocate the
next larger block size (1024K). There are none available.
Any sugestion?
I do not know what determines how much memory ls allocates.
I have similar memory allocation failures on my system. It does not appear
to use a shared uclibc. I do not know (yet) how to enable building a shared
uclibc or how to change the builds to use it. If the instructions are
straightforward, I would like to hear from someone.
Regards,
Luis
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org <mailto:uClinux-dev at uclinux.org>
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org <mailto:uclinux-dev at uclinux.org>
http://mailman.uclinux.org/mailman/options/uclinux-dev
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov <mailto:baker at usgs.gov>
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org <mailto:uClinux-dev at uclinux.org>
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org <mailto:uclinux-dev at uclinux.org>
http://mailman.uclinux.org/mailman/options/uclinux-dev
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com <mailto:gerg at snapgear.com>
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close FAX: +61 7 3217 5323
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com <mailto:gerg at snapgear.com>
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close, FAX: +61 7 3891 3630
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov <mailto:baker at usgs.gov>
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close FAX: +61 7 3217 5323
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
Larry Baker
2012-10-31 18:24:24 UTC
Permalink
Post by Greg Ungerer
Hi Larry,
Post by Larry Baker
Post by Luis Alves
Post by Luis Alves
Post by Greg Ungerer
What you are seeing though is a classic case of memory fragmentation.
Thats true. If I spam a lot of 'ls' with other commands between, it
eventually works. (I guess at some point a memory chunk of 1024k
becomes available).
I've tried to rebuild the kernel using as memory allocator SLOB and
SLUB, but the result seems to be the same as using SLAB.
I've tried those solutions too -- no good. I decided that was because those are kernel memory allocation policies, and have no effect on user memory allocation. Is that correct?
Post by Luis Alves
Post by Luis Alves
Post by Greg Ungerer
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
I've tried with standalone 'ls' and the result is the same.
It also fails wanting to allocate exacly 528384 (always the same value
and only when ls'ing NFS mounts).
(the mem alloc value is actually pretty rounded 512k + 4k)
Could it be related to NFS kernel code?
[<008446e8>] 0x8446e8 (inside <warn_alloc_failed>)
[<0084683a>] 0x84683a (inside <__alloc_pages_nodemask>)
[<00851e34>] 0x851e34 (inside <do_mmap_pgoff>)
[<008526c2>] 0x8526c2 <sys_old_mmap>
[<008526c2>] 0x8526c2 <sys_old_mmap>
[<0084e200>] 0x84e200 (inside <vm_mmap_pgoff>)
[<0085270a>] 0x85270a (inside <sys_old_mmap>)
That looks like an mmap system call. And mmap is the underlying
allocator used by malloc() in uClibc (for non-MMU). So to me it
looks like a malloc somewhere in busybox. That is just a first
guess, it needs some further investigation.
Post by Luis Alves
Don't know if it's relevant, but I'm using latest kernel sources from
your git, gcc-4.7.2, running kernel from flash (XIP) and user binaries
compiled with -msep-data (running from flash also).
I wouldn't suspect anything different. But if you can try older kernel
versions you could confirm that.
Post by Luis Alves
By the way, like Larry pointed out, is there a way to compile user
binaries with shared flat libraries?
Long time ago I've tried to enable UCLIBC_FORMAT_SHARED_FLAT in uClibc
config, but then it was failing building (don't remember where/why it
fails).
I haven't used shared libraries in these setups for years. So I am
not sure what state they are in. Seems like it might have bit rotted
from your experiences.
Rats.
It seems to me the root of the problem on memory constrained systems is the power-of-2 memory allocator; a boxcar memory allocator would work better, I think. I have read there used to be a non-power-of-2 memory allocator, named page_alloc2. Is there any chance it can be revived for 2.6 kernels? (See my post of 23 October 2012.)
I am sure it could be. But it would take a little effort though.
I think Phil Wilshire made an effort a few years back to bring the
page_alloc2 code up to date with 2.6 kernels.
Yeah, I found Phil back in March, before I found the uClinux.org mailing list (e.g., David McCullough's 1 Aug 2007 posting).
Post by Greg Ungerer
HI Larry,
I gave up work on that allocator quite a while ago !!!
There are other options available now
Slub for example.
Best Regards
Phil Wilshire
No good juju there.
Post by Greg Ungerer
Google and check the
archives you may be able to find it.
Whose archives? kernel.org? (Was page_alloc2 part of the mainline 2.4 kernel distribution? under arch?) uclinux.org? Am I looking for page_alloc2, or a different name? Any advice on what changes to look for to port the 2.4 code to 2.6?
Post by Greg Ungerer
Regards
Greg
Post by Larry Baker
Post by Luis Alves
Post by Luis Alves
Post by Greg Ungerer
Hi Luis,
Post by Luis Alves
Thanks for the explanation Larry.
Meanwhile I've read some stuff about non-MMU memory alocation (I had
the wrong idea of how memory was allocated).
I guess my system ram is not that much (only 8Mb)...
What you are seeing though is a classic case of memory fragmentation.
You have eneough free RAM in total, but not enough in s ingle
contiguous block. On a paged MMU system this is not a problem, pages
can be mapped so you see a contiguous block the size you want. You
can't do that without no MMU and paging.
Post by Luis Alves
How would shared libs solve the problem?
The problem is allocating memory for 'ls' data and not 'ls' itself. Is
this correct?
I would dig into the busybox ls code. Wanting to allocate a block of
512k in a single chunk seems a bit crazy.
Regards
Greg
Post by Luis Alves
Post by Luis Alves
Luis,
DMA: 36*4kB 30*8kB 29*16kB 21*32kB 6*64kB 7*128kB 4*256kB 3*512kB
0*1024kB 0*2048kB 0*4096kB = 5360kB
The largest block of memory your system has available is 524288 (512*1024).
Allocation of length 528384 from process 94 (ls) failed
The requested size is larger than that, so uClinux tries to allocate the
next larger block size (1024K). There are none available.
Any sugestion?
I do not know what determines how much memory ls allocates.
I have similar memory allocation failures on my system. It does not appear
to use a shared uclibc. I do not know (yet) how to enable building a shared
uclibc or how to change the builds to use it. If the instructions are
straightforward, I would like to hear from someone.
Regards,
Luis
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org <mailto:uClinux-dev at uclinux.org>
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org <mailto:uclinux-dev at uclinux.org>
http://mailman.uclinux.org/mailman/options/uclinux-dev
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov <mailto:baker at usgs.gov>
_______________________________________________
uClinux-dev mailing list
uClinux-dev at uclinux.org <mailto:uClinux-dev at uclinux.org>
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev at uclinux.org <mailto:uclinux-dev at uclinux.org>
http://mailman.uclinux.org/mailman/options/uclinux-dev
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com <mailto:gerg at snapgear.com>
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close FAX: +61 7 3217 5323
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com <mailto:gerg at snapgear.com>
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close, FAX: +61 7 3891 3630
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov <mailto:baker at usgs.gov>
--
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close FAX: +61 7 3217 5323
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
Larry Baker
US Geological Survey
650-329-5608
baker at usgs.gov
Greg Ungerer
2012-11-01 06:42:26 UTC
Permalink
Hi Larry,
[snip]
Post by Larry Baker
Post by Greg Ungerer
Post by Larry Baker
It seems to me the root of the problem on memory constrained systems is the power-of-2 memory allocator; a boxcar memory allocator would work better, I think. I have read there used to be a non-power-of-2 memory allocator, named page_alloc2. Is there any chance it can be revived for 2.6 kernels? (See my post of 23 October 2012.)
I am sure it could be. But it would take a little effort though.
I think Phil Wilshire made an effort a few years back to bring the
page_alloc2 code up to date with 2.6 kernels.
Yeah, I found Phil back in March, before I found the uClinux.org mailing list (e.g., David McCullough's 1 Aug 2007 posting).
Post by Greg Ungerer
HI Larry,
I gave up work on that allocator quite a while ago !!!
There are other options available now
Slub for example.
Best Regards
Phil Wilshire
No good juju there.
Post by Greg Ungerer
Google and check the
archives you may be able to find it.
Whose archives?
My thinking was the uclinux.org email archives.
Post by Larry Baker
kernel.org? (Was page_alloc2 part of the mainline 2.4 kernel distribution? under arch?) uclinux.org? Am I looking for page_alloc2, or a different name? Any advice on what changes to look for to port the 2.4 code to 2.6?
page_alloc2 was never in 2.4 mainline kernels - but nothing
related to uClinux (non-MMU) was. It was a part (and still is) of
the uclinux.org sources. You can find it in pretty much any version
of the uclinux-dist (even the most recent ones) as:

uClinux-dist/linux-2.4.x/mmnommu/page_alloc2.c

I haven't looked at it in quite a while, really no idea how difficult
it will be to bring up to date.

Regards
Greg


------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg at snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
8 Gardner Close FAX: +61 7 3217 5323
Milton, QLD, 4064, Australia WEB: http://www.SnapGear.com
Loading...