gujin-devel Mailing List for gujin boot/system loader

Etienne Lorrain <eti...@ya...> writes:

>  --- "Eric W. Biederman" <ebi...@xm...> a =E9crit=A0:
> > Etienne Lorrain <eti...@ya...> writes:
> > >  If someone wants to change it completely, he will
> > >  have to rewrite tools which are accessing this structure (rdev) and
> > >  also the bootloaders which are setting up fields into it already.
> > >  This will involve re-coding real-mode i8086 assembly, and there is l=
ess
> > >  and less people knowing how to do it.
> >=20
> > True.  However for the moment I am up for making this kind of change.
> > rdev needs to be recoded anyway as it is not portable.  We need a rdev
> > like tool to attach a command line and a ramdisk to arbitrary linux
> > kernels.
>=20
>   I am from the old Unix school, I like text - so IMHO this could go in
>  a string, like "root=3D/dev/hda3 initrd=3Dinitrd-3.gz". That is why I pr=
oposed
>  to put that string in the "comment" field of the gzip file (see RFC).
>  Unfortunately right now there is no tool to set up this field, but all t=
he
>  uncompression tools seems to treat correctly (skip) this field if presen=
t.
>  Is there someone against it? Is there a software (like the "znote"
>  for pkzip) available on Internet to set the comment field of a gz file?

If you are going with gzip.  That definentily sounds like a good strategy.
=20
> > >  My main problem is the order the things are done: First load compres=
sed
> > >  files at defined addresses, then call a kernel function which callba=
ck
> > >  the BIOS, then uncompress files.
> >=20
> > How is this a problem.  Is it the error handling or something else?
>=20
>  I hate to destroy random memory content and then ask to the lower
>  layers to behave correctly. For instance I used a DOS debugger to find
>  some of my bugs - it is not possible if the memory at 0x100000 has been
>  overwritten. Even simply my logger/debugger (writing information in a
>  log file under DOS) can not work.

There is sense it that.  Having a way to do all reasonable of the
error checking up front and then going forward with running the kernel.

> > I like the fact that the kernel decompresses itself.  This is one less
> > thing that the bootloader doesn't have to do.  And allows the
> > compression scheme to be changed at least theoretically.
>=20
>  I need the decompressor. People will want icons to clic on for Linux,
>  and I plan to uncompress them with my gunzip. Maybe one day I will even
>  compress a part of my bootloader, for instance the different language
>  messages or even part of the code. Note that this ungzip-er is very smal=
l,
>  5 Kb for the one in 0.4, 4.5 Kb for the current one - and in some cases
>  do not even need a 32 Kb memory window, unlike the real zlib.
>  Keep in mind that this bootloader may one day go to the EEPROM of a
>  network card (32 Kb max) or in the empty FLASH area of the BIOS...
>  Also have a look at the memory requirement of bzip2, few Mb...

O.k.  I'll definentily have to have a look at it then.  The gunzip code
I have seems to have been larger.
=20
> > Why does Gujin need to do the decompression?  Is this simply for
> > better error checking, and error handling?
>=20
>  Also, the CRC32 calculation is nearly the only way to guaranty that you
>  have a real kernel - and did not do any error in loading the file. The
>  usual problems I am waiting here is to load wrong sectors because I have
>  a bug in the CHS description of the disk, or I have read a corrupted sec=
tor
>  because the floppy had bad sectors and the ECC correction did a bad job.

O.k.  That is one area where you are definentily ahead of me.  I have
been playing with some kind of checksum but I haven't yet.
=20
> > >  Once Gujin has started, I have a complete C environment so I can load
> > >  files, treat errors, display messages. I can do this either from cold
> > >  boot or from DOS (think of the first install of Linux on a system).
> >=20
> > This definentily sounds nice.  Do you envision the setup code
> > returning to the bootloader if a fatal error occurs?
>=20
>   Even more, I envision running GPL maintenance software - and return to
>  the bootloader at the end because this software did take care to not
>  modify memory below 1Mb.
>  For instance I have corrected a small bug in the bootloader (first stage)
>  because I had a sector to read on the HD - and this sector cannot be read
>  at the first time (needs a 0x20 - read with retry - IDE command,
>  a 0x21 - read without retry - return "uncorrected ECC error") and I was
>  not waiting enough for the ECC correction to finnish its task.
>  Because one of the main feature of e2fs is to not fragment files, you
>  will never run an unfragmenter, so there will (probably) be more and
>  more sectors which will need ECC correction each time they are read.
>  What is then needed is a small application displaying the ratio of
>  block needing ECC correction and maybe just read and rewrite them
>  to optimise the system.
>=20
> > >  This interface would still not handle a distribution where there is
> > >  few kernel files:
> > >  /boot/Linux-2.4.8-SMP
> > >  /boot/Linux-2.4.8-UP
> > >  /boot/Linux-2.4.8-386
> > >  /boot/Linux-2.4.8-Pentium
> > >  And the bootloader should just select automagically the right kernel.
> >=20
> > At this port I have linux booting from linux so what I would do is
> > load /boot/Linux-2.4.8-i386 and then from linux select the right
> > kernel modify the bootloader parameters, and boot the optimal kernel.
>=20
>   IHMO a bit complex, managing the command line parameters for the two
>  kernels (like ide addresses, scsi disk order...)

Oh it can get complex, but if all you want to do is to invoke another
kernel with exactly the same options you were invoked with it is a
simple thing to do.

> > >  Moreover, going from a simple solution (loading the binary image of
> > >  an ELF file) to a complex one (as described) to solve problem which
> > >  may appear in the future is not my way of thinking: it is already
> > >  complex enought to do simple software.
> >=20
> > I believe it can be done much more simply.  And I think I can make the
> > time to get it done, over the long term.   I can do with an ELF
> > binary, not it's binary ala objdump -O binary I can do what you
> > propose to do with concatenated GZIP files.  And with that I don't
> > need decompression code in the bootloader.=20
>=20
>   For me, I choose to load a GZIP because its format is trivial enough
>  to extract, and a "objdump -O binary" because I do not want to do the
>  loading part of an ELF file.=20

For purposes of a loading a binary the ELF program header simply says load
this chunk of file to this chunk of memory.  So while it is a touch more
complex then a GZIP header it is simpler than a whole gzip file.

With ELF I have so far implemented 3 bootloaders, that can handle it
which allows me to boot arbitrary kernels.

> GZIP is also used for initrd, so you need
>  it if you want to check the integrity what you have loaded.
>   You can try any other method you want, I can't say I never did a
>  wrong choice.

Where I have been focusing is on network booting, and booting in small
environments without a BIOS.  And for those having the option to
package the kernel and ramdisk, and the command line together is
important, as is the option to reduce size by using a different
compression algorithm.  Which is my primary reason for wanting to
include the decompressor with the kernel, and require the bootloader
to use it.


That isn't all of my comments but I need to run off to work now.

Later,
Eric

 --- "Eric W. Biederman" <ebi...@xm...> a écrit :
> Etienne Lorrain <eti...@ya...> writes:
> >  If someone wants to change it completely, he will
> >  have to rewrite tools which are accessing this structure (rdev) and
> >  also the bootloaders which are setting up fields into it already.
> >  This will involve re-coding real-mode i8086 assembly, and there is less
> >  and less people knowing how to do it.
> 
> True.  However for the moment I am up for making this kind of change.
> rdev needs to be recoded anyway as it is not portable.  We need a rdev
> like tool to attach a command line and a ramdisk to arbitrary linux
> kernels.

  I am from the old Unix school, I like text - so IMHO this could go in
 a string, like "root=/dev/hda3 initrd=initrd-3.gz". That is why I proposed
 to put that string in the "comment" field of the gzip file (see RFC).
 Unfortunately right now there is no tool to set up this field, but all the
 uncompression tools seems to treat correctly (skip) this field if present.
 Is there someone against it? Is there a software (like the "znote"
 for pkzip) available on Internet to set the comment field of a gz file?

> >  My main problem is the order the things are done: First load compressed
> >  files at defined addresses, then call a kernel function which callback
> >  the BIOS, then uncompress files.
> 
> How is this a problem.  Is it the error handling or something else?

 I hate to destroy random memory content and then ask to the lower
 layers to behave correctly. For instance I used a DOS debugger to find
 some of my bugs - it is not possible if the memory at 0x100000 has been
 overwritten. Even simply my logger/debugger (writing information in a
 log file under DOS) can not work.

> I like the fact that the kernel decompresses itself.  This is one less
> thing that the bootloader doesn't have to do.  And allows the
> compression scheme to be changed at least theoretically.

 I need the decompressor. People will want icons to clic on for Linux,
 and I plan to uncompress them with my gunzip. Maybe one day I will even
 compress a part of my bootloader, for instance the different language
 messages or even part of the code. Note that this ungzip-er is very small,
 5 Kb for the one in 0.4, 4.5 Kb for the current one - and in some cases
 do not even need a 32 Kb memory window, unlike the real zlib.
 Keep in mind that this bootloader may one day go to the EEPROM of a
 network card (32 Kb max) or in the empty FLASH area of the BIOS...
 Also have a look at the memory requirement of bzip2, few Mb...

> Why does Gujin need to do the decompression?  Is this simply for
> better error checking, and error handling?

 Also, the CRC32 calculation is nearly the only way to guaranty that you
 have a real kernel - and did not do any error in loading the file. The
 usual problems I am waiting here is to load wrong sectors because I have
 a bug in the CHS description of the disk, or I have read a corrupted sector
 because the floppy had bad sectors and the ECC correction did a bad job.

> >  Once Gujin has started, I have a complete C environment so I can load
> >  files, treat errors, display messages. I can do this either from cold
> >  boot or from DOS (think of the first install of Linux on a system).
> 
> This definentily sounds nice.  Do you envision the setup code
> returning to the bootloader if a fatal error occurs?

  Even more, I envision running GPL maintenance software - and return to
 the bootloader at the end because this software did take care to not
 modify memory below 1Mb.
 For instance I have corrected a small bug in the bootloader (first stage)
 because I had a sector to read on the HD - and this sector cannot be read
 at the first time (needs a 0x20 - read with retry - IDE command,
 a 0x21 - read without retry - return "uncorrected ECC error") and I was
 not waiting enough for the ECC correction to finnish its task.
 Because one of the main feature of e2fs is to not fragment files, you
 will never run an unfragmenter, so there will (probably) be more and
 more sectors which will need ECC correction each time they are read.
 What is then needed is a small application displaying the ratio of
 block needing ECC correction and maybe just read and rewrite them
 to optimise the system.

> >  This interface would still not handle a distribution where there is
> >  few kernel files:
> >  /boot/Linux-2.4.8-SMP
> >  /boot/Linux-2.4.8-UP
> >  /boot/Linux-2.4.8-386
> >  /boot/Linux-2.4.8-Pentium
> >  And the bootloader should just select automagically the right kernel.
> 
> At this port I have linux booting from linux so what I would do is
> load /boot/Linux-2.4.8-i386 and then from linux select the right
> kernel modify the bootloader parameters, and boot the optimal kernel.

  IHMO a bit complex, managing the command line parameters for the two
 kernels (like ide addresses, scsi disk order...)

> >  Moreover, going from a simple solution (loading the binary image of
> >  an ELF file) to a complex one (as described) to solve problem which
> >  may appear in the future is not my way of thinking: it is already
> >  complex enought to do simple software.
> 
> I believe it can be done much more simply.  And I think I can make the
> time to get it done, over the long term.   I can do with an ELF
> binary, not it's binary ala objdump -O binary I can do what you
> propose to do with concatenated GZIP files.  And with that I don't
> need decompression code in the bootloader. 

  For me, I choose to load a GZIP because its format is trivial enough
 to extract, and a "objdump -O binary" because I do not want to do the
 loading part of an ELF file. GZIP is also used for initrd, so you need
 it if you want to check the integrity what you have loaded.
  You can try any other method you want, I can't say I never did a
 wrong choice.
 
  Etienne.


___________________________________________________________
Do You Yahoo!? -- Vos albums photos en ligne, 
Yahoo! Photos : http://fr.photos.yahoo.com

 --- Giacomo Catenazzi <ca...@ma...> a écrit :
> Etienne Lorrain wrote:
> >  These two files in memory have also to be at fixed linear
> >  addresses in real mode - and if you have a memory manager
> >  (himem.sys) loaded, these address may not be free. Usually
> >  you will find at the bottom of the himem memory the smartdrv
> >  (disk cache) data. It is then impossible to load a file at a random
> >  memory address and stay in real mode to do futher processing.
> >  In this case, Gujin is just malloc'ing the memory (using himem.sys),
> >  loading and decompressing this file (checking its CRC32), and
> >  only then disable interrupts, switch to protected mode, copy
> >  the file at its intended linear address and jump to the kernel
> >  code.
> > 
> 
> hmm.
> 
> You say: BIOS/hardware can be broken, let use DOS do load Linux!

  There are ISA/PCI cards which are initialised before the first sector
 of the HD/FD is loaded (SCSI, network boot ROM or special devices), but
 it is true that they are not usually using memory at 0x100000.
  I have just done this relocation thinking that it may be nice to be
 able to load a kernel over the 16 Mbytes limit - to free a maximum
 of space to the 16 Mb DMA-able area (if DMA is 16 Mb only) - but that
 is stuff for the future because the kernel would be linked at another
 address. I applied this late-relocation in case of himem.

> IMHO using DOS (himem.sys and letting DOS to setting our
> hardware in ibmbio.com and ibmdos.com) give us more problem
> that solution!

  I am not able to open the A20 gate in all the ways himem.sys can,
 objdump will be your friend here. Note that Linux has never booted
 on such a special machine - using I/O ports 0x329 or 0x320 to open A20 -
 I just leave the door open to run from DOS/win, even for beginner
 who do not know what is himem/emm386 - or people wanting to remote
 boot Linux from a DOS/win machine getting vmlinuz from the network.

> We should have complete control to hardware, not letting DOS
> to hide/modify the BIOS segment 0040:0000 and some other
> hardware setting.

  So Gujin will boot from an IDE HD not supported by the BIOS,
 on a VGA card without BIOS (or nearly: see VGA_MEMORY compile switch).
 It can also do what loadlin does - I do not want to maintain two
 identical software.

> Thus we should (if possible) use only BIOS call (or directly hardware),
> but forget DOS. (BTW you know what DOS makes before himem.sys ?
> Do we have the sources?)

  Forget DOS/Win - what a dream... Can I join your dream?

  Etienne.

___________________________________________________________
Do You Yahoo!? -- Vos albums photos en ligne, 
Yahoo! Photos : http://fr.photos.yahoo.com

Etienne Lorrain wrote:
 
>  These two files in memory have also to be at fixed linear
>  addresses in real mode - and if you have a memory manager
>  (himem.sys) loaded, these address may not be free. Usually
>  you will find at the bottom of the himem memory the smartdrv
>  (disk cache) data. It is then impossible to load a file at a random
>  memory address and stay in real mode to do futher processing.
>  In this case, Gujin is just malloc'ing the memory (using himem.sys),
>  loading and decompressing this file (checking its CRC32), and
>  only then disable interrupts, switch to protected mode, copy
>  the file at its intended linear address and jump to the kernel
>  code.
> 

hmm.

You say: BIOS/hardware can be broken, let use DOS do load Linux!

IMHO using DOS (himem.sys and letting DOS to setting our
hardware in ibmbio.com and ibmdos.com) give us more problem
that solution!

We should have complete control to hardware, not letting DOS
to hide/modify the BIOS segment 0040:0000 and some other
hardware setting.
Thus we should (if possible) use only BIOS call (or directly hardware),
but forget DOS. (BTW you know what DOS makes before himem.sys ?
Do we have the sources?)


	giacomo

hpa wrote:
> Etienne wrote:
> >  - Loads Linux kernel images (zImage and bZimage) without the help
> >    of LILO, and short-circuit all real-mode code in the kernel to
> >    start at the first protected mode instruction of the kernel.
>
> This is a very bad idea.  The kernel entry point is in real mode for a
> reason: it means that the kernel doesn't have to rely on the boot
> loader to provide the services it needs from real mode before
> entering protected mode once and for all.  The interface to the real
> mode entry point is narrow and stable, the protected mode entrypoint
> is a kernel internal and doesn't have an interface that is guaranteed
> to be stable -- again, for good reason.

 Thanks for the constructive criticism, now it is time to explain
 why I did it this way - it is far to be the simplest solution.

 First, I need to say that the Linux current interface is really
 stable: if you define "BOOT_KERNEL_BEFORE_2_0" in vmlinuz.[ch],
 Gujin will be able to boot pre- 1.0 Linux versions. Also, Gujin
 has always booted kernels this way.

 Considering the kernel interface to the bootloader, I find it is
 IMHO a bit complex: you not only have to give parameters in
 memory (at 0x9000:0000 or at %esi on 2.4.1+) but also provide
 callbacks (BIOS calls) for quite a lot of things.

 Alan Cox described it (incompletely) in his message in june,
 subject "draft BIOS use document for the kernel" at:
http://marc.theaimsgroup.com/?l=linux-kernel&m=99322719013363&w=2

 You also have to put in memory one or two compressed files
 (kernel and initrd) without any way to check if those hundred
 of Kbytes have been correctly read and are not corrupted.
 There will be no way - if an error is detected by the legacy
 loader at the decompression stage - to return to the bootloader
 saying: Fatal error, please select another kernel.

 These two files in memory have also to be at fixed linear
 addresses in real mode - and if you have a memory manager
 (himem.sys) loaded, these address may not be free. Usually
 you will find at the bottom of the himem memory the smartdrv
 (disk cache) data. It is then impossible to load a file at a random
 memory address and stay in real mode to do futher processing.
 In this case, Gujin is just malloc'ing the memory (using himem.sys),
 loading and decompressing this file (checking its CRC32), and
 only then disable interrupts, switch to protected mode, copy
 the file at its intended linear address and jump to the kernel
 code.

 Then, with the "old" method to load the kernel, you have the video
 selection menu. Well, it is a bit complex a thing to do in
 a bootloader, and its interface is complex (not speaking of the
 user interface), considering that you have already written everywhere
 in memory without knowing if (for instance) a video driver was loaded.
 All the video selection stuff has be to handled before selecting
 a kernel to load, IMO.

 I could probably find other problem, but I do not want to do
 destructive criticism.

What do I propose? Lets put is simply:

 - The kernel file is a simple binary image to load at 0x100000,
 so you get it from the linux ELF file by just doing:
  objcopy -O binary -R .note -R .comment -S /usr/src/linux/vmlinux kernel
  gzip -9 kernel
 There is absolutely no limit on its size.

 - The initrd (if present) is a gzip of a filesystem image.

 - All information needed by the kernel is set in memory. It is
 clearly described on a C structure (vmlinuz.h::struct linux_param).
 This structure contains also (for reference) old fields which
 are no more used, or fields which were used only by previous
 bootloader. The is _no_ BIOS callback.

 - The hardware is setup correctly, for instance for the current
 video mode - if you are in graphic 8 BPP - the 256 color palette
 is set up with reasonnable values independant of the video card.
 Basically, hardware is immediately useable by the kernel.

 - There is a compatibility mode to load vmlinuz/zImage/bzImage.

 Note that all this is coded and working (I hope so). I know that
 with this solution, the bootloader and the kernel are more
 linked together, but I know also that the bootloader has to
 be a lot more intelligent (considering the number of related
 messages in the Linux lists). It should not try to load a Pentium
 compiled kernel on a 486. It should setup the video card (so
 be ready to get "invalid instruction exceptions" while executing
 BIOS calls). It should not try to run a corrupted kernel. It should
 not crash if a kernel/initrd file has been moved or two hard disks
 have been exchanged. It should display clear messages on what is
 wrong.
 It is up to the bootloader to crash if one BIOS call modify one
 register which is documented as constant (if it did not take care)
 - and in DEBUG mode send some informations to a serial/parallel port.
 When a new buggy BIOS appear, it is up to the bootloader to be upgraded,
 not to the Linux kernel.
 In short it is a real and complete software which handle all/most
 of the (buggy) BIOSes in their natural environment: the i386 real mode.
 If there is few fields to add to "struct linux_param" (really
 unusual from history), then Gujin will be upgraded. Anyway it is
 GPL - and written in C so there is a lot of people around able to
 change it, unlike assembly code.

 It is not as clean as described because of the APM / ACPI /
 PCI configuration BIOS32 calls; but that did not change by the
 Gujin bootloader. Note also that the SMP tables are passed in
 memory.

 One last thing I have to add (for people who have read up to here),
 is that having removed the "header" of the vmlinuz file, I lack the
 only important parameter in this header: the root device (rdev command).
 Right now, I am guessing it in a lot of configurations, but that
 is not a perfect solution. I think one of the simplest way is
 to add a "root=/dev/hd**" in the described comment field of the GZIP
 header; that is still not coded.
 From the same area, there is no way to know if a kernel will be able
 to boot in graphic mode (support of VESA framebuffer and which BPP
 are available). If vesafb is not compiled in and you start the
 kernel in graphic, the kernel boots but the display is like a
 crash... Right now the dirty way is the write-protect bit of the
 vmlinuz file - not a long term solution.

  Comment anyone?

  Etienne.


___________________________________________________________
Do You Yahoo!? -- Vos albums photos en ligne, 
Yahoo! Photos : http://fr.photos.yahoo.com