docutils-develop Mailing List for Docutils: Documentation Utilities (Page 418)

On Mon, Aug 26, 2002, David Goodger wrote:
> 
> I don't mind at all.  Actually, I prefer the discussions online; it
> opens up the possibility for more input.

this is relevant to me, but can't talk about it with my broken wrist.
-- 
Aahz (aa...@py...)           <*>         http://www.pythoncraft.com/

Project Vote Smart: http://www.vote-smart.org/

I'd already begun a follow-up to my last message, which I'll
interweave into this reply.

[Dethe]
> Thanks for the great feedback.  I hope you don't mind me taking this
> back online, I think there's some valuable stuff here.

I don't mind at all.  Actually, I prefer the discussions online; it
opens up the possibility for more input.

[David]
>> I just hope I don't scare you away.
> 
> Not a bit of it.

Glad to hear it.  In the message I'd already begun, I wrote:

    If this small task has grown too big or or time-consuming or
    onerous for you to handle, please let me know and I'll take over,
    with gratitude for work done and ideas inspired.

So you've got an escape clause if you change your mind. ;-)

>> I'm looking at the "utils.cheapDirective" function you wrote.  I
>> think it's a very good idea to provide a generic directive parser
>> function, but to be truly useful, this function needs to be even
>> *more* generic.  Please bear with me here.
>
> More generic would be good, I was working out from the images
> directive and trying to factor what was specfic to that directive
> from what was common to all directives as I went.

And a very fruitful side-trip it has become!  It's a step I hadn't
taken.  Every new directive began with a copy & paste from the most
similar existing one, but I hadn't looked at the patterns yet.

>> From the markup spec, there are 3 physical parts to a directive:
>> 
>> 1. Directive type (identifier before the "::")
>> 2. Directive data (remainder of the first line)
>> 3. Directive block (indented text following the first line)
> 
> Except that there is a fourth part: The attribute list.

That's a *logical* part, not physical (at least, not by the original
definition).  No matter; I think we'll dispense with the old
"physical" terms as obsolete.

>> I think the "include" directive should be reStructuredText-only,
>> and the "raw" directive should have an optional second argument, a
>> path.  They definitely should not duplicate each other's
>> functionality (TOOWTDI).  In the terms defined above,
> 
> I'm not disagreeing, but I got a bit lost in the debate over :raw:
> vs. :include: and wanted a) some context to help me form an
> opinion, and b) to test the generalized directive parsing by using
> it for more than one directive.

Sure.  That's cool.

>> - "include" should have one argument, a path. (1)
>> 
>> - "raw" should have one or two arguments -- a format ("html" etc.)
>>   and an optional path -- and content, but only if there was no
>>   path argument.  So this would be either a (1) or a (1,3)
>>   directive.  So that forces us to split the parse in two, perhaps
>>   into "parse_directive" (which parses the arguments & attributes),
>>   plus "get_directive_content".
> 
> Not necessarily.  Is there any reason we can't parse a directive and
> return a 3-tuple (arguments, options, content).  It would then be up
> to the individual directive to test that these do have values, if
> required.

Originally, I was thinking that if a directive didn't *need* a content
block, it shouldn't consume it any that happened to be there.  That
would allow a directive to be followed by a block quote, but it makes
directives conceptually more difficult than they need to be.  While
updating the markup and directive specs as per my last message, I
thought some more about the logical parts of a directive, and I now
believe that the original thinking was wrong.  Instead, all indented
text following a directive *should* be consumed by the directive.  The
"parse_directive" function should signal an error if there *is*
content, but the directive doesn't ask for it.  If a block quote
should follow a directive, an empty comment inserted between them will
do the trick.  Docs updated accordingly.

And yes, you're right about returning a uniform set of data.  But we
actually need a 4-tuple; read on.

>> Note that there are no attributes/options now.  If an attribute is
>> required, it shouldn't be an attribute.  It's analogous to
>> command-line options: "required option" is an oxymoron.  I'm thinking
>> of changing the terminology in the spec from "attribute" to "option"
>> to help reinforce this.

(As I threatened here, I've changed terminology from "directive
attributes" to "directive options", in the docs and in the parser
code.  See the latest CVS or snapshot.)

> OK, but this means sub-parsing the arguments instead of pulling data
> out of the attributes.  My preference would be to have a standard
> way to specify not only the types of attributes, but default values
> for them.  Then there are no optional attributes, just default or
> explicitly set.  This is a more XML-ish way to go.

The concept of "optional" is useful.  I'm using the XML idea of
#IMPLIED attributes, rather than defaults in the DTD (or in the
attribute parsing code, in our case).  I've always found it more
flexible for the downstream parts of the processing chain to make the
decisions; keep your options open.  If you put in default values
early, you lose the information that the attribute just *wasn't
specified*, and that can be valuable information lost.

Modelling directives on shell commands works well.  Let's go with it.

>> Looking at the directives in the test1.rst file in order, first
>> there's::
>> 
>>     .. include:: file:test2.rst
>> 
>> I'm not comfortable with URL syntax here.  I really think it's
>> YAGNI, and may open up a big can of worms.  So that one should
>> become::
>> 
>>     .. include:: test2.rst
> 
> Well, I think we *do* need URIs if we want includes to be useful to
> ZReST (which is not file-system based, but lives in an
> object-oriented database accessible by URI).  Obviously, we don't
> need a file: URI for such a trivial (and file-based) example, that's
> just a way to test that URIs work in general.

OK, good use case.  But I'm still uncomfortable with the "openAny"
function you sent::

    def openAny(path):
        try:
          # is it a file?
          return open(path)
        except :
          try:
            # is it a url?
            return urlopen(path)
          except (URLError, ValueError):
            # treat as a string
            return StringIO(path)

Especially the final StringIO part; the function should simply fail.
Should we check if "path" is a URL first, to avoid the "open(path)"
failure?  Or is this a case of "look before you leap" vs. "it's easier
to ask forgiveness than permission"?  Can a URI look like a filesystem
path?  Seems a bit ambiguous to me.

Out of curiosity, what would a Zope/ZReST URL look like?

>> The next one is an "include"/"raw" hybrid::
>> 
>>     .. include:: test3.rst
>>        :raw: 
>>        :format: html
> 
> Well, there were a few possibilities tossed around in email.  I
> couldn't remember why the format was even needed (it's *raw*, what
> else do we need?), but then my data wouldn't show up using html.py
> and I had to dig through the writer code to figure out that raw
> nodes are tested for format type.

Yes, it wouldn't do much good to insert raw HTML in the middle of PDF.
The "raw" directive is meant to be a solution of last resort anyhow;
it's not portable.

>> It should be::
>> 
>>     .. raw:: html test3.rst
> 
> OK, if that's the correct way to do it in reST, I'll do that.  I
> still prefer the default attributes method discussed above rather
> than multiple arguments because it makes the names and types of the
> argument/option/attribute explicit.

Discussed further below.

>> I noticed that you've got the attributes *after* the content in the
>> next ones::
>> 
>>     .. include::
>>        This is a <super>Test</super> of the <sub>Emergency</sub>
>>        <strike>Broadcasting</strike> &quot;System&quot;
>>        :raw:
>>        :format: html
> 
> Yes, that was my mis-parsing of the directive (because I was working
> from images.py, not the spec).

Aha!

> The way I was grabbing (or failing to grab?) the content threw an
> exception if I put the attributes first, but worked OK if I put them
> after.  I didn't like this either.

The spec is there for a reason. :-) But it's not immutable.  It can be
changed when there's good reason.  The code too; there's a *reason*
we're not at release 1.0 yet!  This is a learning experience.

>> Attributes always come *before* the content (who's to say that
>> ":format: html" isn't valid raw data in some format?).  In any case,
>> that directive should become::
>> 
>>     .. raw:: html
> 
> OK.  But the same arguments apply.  Who's to say :format: html isn't
> valid raw data at the beginning of the data as well as easily as at
> the end?  I agree that it's aethetically and functionally better to
> have the attributes first, just not because the string ":format:
> html" might appear in the data.

That's why the blank line is necessary between options and content.

> Also, the problem with using multiple comes up. Arguments to
> :raw: are 
> 
> path format
> or
> format
> 
> But it seems more intuitive to me to put the required argument
> first: so the first argument to :raw: would always be format and we
> don't have to run tests or special cases.  If there's a second
> argument, then it's path.

You mis-read.  It *is* ".. raw:: format [path]" (the *second*
argument, "path", is optional).

> This still isn't as clear and explicit (to me) as using attributes,
> but better than having argument position be dependent on number of
> arguments.

It could be::

    .. raw:: format
       :source: path/URL

Nothing wrong with that, I suppose.  Since it is an *optional*
argument, it does fit into the "option" mold.  And in fact, it could
be even more explicit (and remove my misgivings at the same time) if
we made the option more specific::

    .. raw:: format
       :file: path

    .. raw:: format
       :url: URL

Although better option names may exist.

>> Next::
>> 
>>     .. raw::
>>        This is <strong>RAW</strong>.  Really, <em>really,</em> raw.
>>        :format: html
>> 
>> Should become::
>> 
>>     .. raw:: html
>> 
>>        This is <strong>RAW</strong>.  Really, <em>really,</em> raw.
>> 
>> (Note the blank line.)
> 
> Oh, there's a blank line *before* the content.  I didn't get that.

Yes.  The directives.txt file was mistaken for the description of
"raw" (although if you didn't *read* it, that shouldn't have mattered
;-).  It's fixed now, and the specs are much more explicit.  See
http://docutils.sf.net/spec/rst/reStructuredText.html#directives, and
http://docutils.sf.net/spec/rst/directives.html.

> I thought the directive *ends* with a blank line.

There was some ambiguity about that, gone now.  The rule is, a
directive block ends with the end of indentation.  That's it.  If a
directive doesn't need a content block, it should be empty, otherwise
it's an error.

> Yikes.  Now I see why you want get_directive_content().  Doesn't
> that introduce all sorts of possible ambiguities?  Shouldn't there
> be one format for directives in all cases so that the users (typing
> in raw text with reST getting in their way as little as possible)
> don't have to remember the special cases for different directives?

Yes and yes, and I came to the same conclusion, as discussed above.
Just "parse_directive()" will be sufficient.

>> The last "raw" directive was::
>> 
>>     .. raw::
>>        :include: test3.rst
>>        :format: html
>> 
>> And should become::
>> 
>>     .. raw:: html test3.rst
>> 
>> (This is the same as the second directive, therefore redundant.)
> 
> The redundancy is deliberate.  I was testing two different
> possibilities and wanted to be sure they both worked.  They do the
> same thing, therefore they work.

Sorry, I should have said, "(This is *now* the same as the second
directive, therefore redundant.)". :-)

> So what you'd like to see is:
> 
> :raw: [filepath] format
> 
> :include: filepath (which must be reST)
> 
> and functions for directives/__init__.py
> 
> parse_directive()
> 
> and 
> 
> get_directive_content()
> 
> right?

Not quite.  Just to be clear, let's summarize:

* ".. include:: filepath" (must be reStructuredText).

* ".. raw:: format" + either:

  - a second, optional "filepath" argument, or
  - a "source" (or equivalent) option, or
  - (perhaps best?) *two* options, one for a filesystem path source,
    the other for a URL source (and we can't use both in one
    directive)

  If the "external source" argument or option is specified (in
  whatever form), there can be no directive content.  If there is,
  it's an error.

* A single function for directives/__init__.py, "parse_directive".
  (Plus any auxiliary functions required, of course.)

* An exception for directives/__init__.py::

     class DirectiveParseError(docutils.ApplicationError): pass

  (It doesn't need an "__init__" method, but it doesn't hurt much.)

The signature for "parse_directive" could be something like this::

    def parse_directive(match, type_name, state, state_machine,
                        option_presets, arguments=None,
                        option_spec={}, content=None):
        """
        Parameters:

        - `match`, `type_name`, state`, `state_machine`, and
          `option_presets`: See `docutils.parsers.rst.directives.__init__`.
        - `arguments`: A 2-tuple of the number of ``(required,
          optional)`` whitespace-separated arguments to parse, or
          ``None`` if no arguments (same as ``(0, 0)``).  If an
          argument may contain whitespace (multiple words), specify
          only one argument (either required or optional); the client
          code must do any context-sensitive parsing.
        - `option_spec`: A dictionary, mapping known option names to
          conversion functions such as `int` or `float`.  ``None`` or
          an empty dict implies no options to parse.
        - `content`: A boolean; true if content is allowed.  Client
          code must handle the case where content is required but not
          supplied (an empty content list will be returned).

        Returns a 4-tuple: list of arguments, dict of options, list of
        strings (content block), and a boolean (blank finish).

        Or raises `DirectiveParseError` with arguments: node (system
        message), boolean (blank finish).
        """

Once "parse_directive" is ready, we'll be able to convert all existing
directives use it.  As a side-effect, we will be able to drop the
"data" parameter from the directive function signature.  The end
result will be much simpler directive code.  Great result!

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

David Goodger wrote:
> 
> Garth Kidd wrote:
> 
> >> So, should it be::
> >>
> >>   .. raw:: html fragment.html
> >>
> >> or::
> >>
> >>   .. include:: fragment.html
> >>      :raw: html
> >
> > +1 to having both both available (i.e. ``raw::`` mapping directly to
> >``include::`` with ``:raw:`` set, or vice versa). 
>
> Normally, the "raw" directive's content will be supplied within the document
> body.  So the question becomes, does the "raw" directive grow an "include"
> attribute (or optional argument), or does the "include" directive grow a
> "raw" attribute?  I'm not sure that "both" is a good answer here.

My first thought after seeing the two syntaxes was that the first
is a shortcut for the second. But, if
  .. raw:: format path
is the same as
  .. include:: path
     :raw: format
why have the raw attribute at all? You can simply have two separate
directives: 'include', which parses as reST, and 'raw', which doesn't.

> > Now, anyone want to explain why it's ``html`` and not a full MIME type?
> > :)
> 
> The "text/" part is implicit? ;-)

I have a feeling a lot of SGML/XML markup formats don't have their own
MIME type--that they're just sent as text/sgml or text/xml. Perhaps
:raw:'s value shouldn't be half a MIME type, but a token used as a
language identifier. One might want to skip printing raw content that
doesn't match the output format, and if there are several different
XML languages, a MIME type won't label the difference.

~fantasai

Hi David,

Thanks for the great feedback.  I hope you don't mind me taking this
back online, I think there's some valuable stuff here.

On Sat, 2002-08-24 at 08:46, David Goodger wrote:
> Dethe Elza wrote:
> > Included are two python files and three test documents.
> 
> Thank you!  It's great to get contributions!
> 
> > I'm getting going on the HOWTO, but having a problem with one of the
> > test cases that I haven't been able to resolve, but probably you can
> > help.
> 
> I'll do my best.  I just hope I don't scare you away.

Not a bit of it.

> > Here's the problem.  If you process test1.rst using the standard
> > html.py it includes test2.rst and processes it, which includes
> > test3.rst.  The content of test3.rst shows up OK, but it isn't being
> > processed as reST.
> 
> I haven't actually run the code yet.  I'll do that next.  In the
> meantime, here are some comments on what you've sent:
> 
> > There's a util.py file which gives some helpers for creating
> > directives
> 
> I'm looking at the "utils.cheapDirective" function you wrote.  I think
> it's a very good idea to provide a generic directive parser function,
> but to be truly useful, this function needs to be even *more* generic.
> Please bear with me here.

More generic would be good, I was working out from the images directive
and trying to factor what was specfic to that directive from what was
common to all directives as I went.

> From the markup spec, there are 3 physical parts to a directive:
> 
> 1. Directive type (identifier before the "::")
> 2. Directive data (remainder of the first line)
> 3. Directive block (indented text following the first line)

Except that there is a fourth part: The attribute list.

> These correspond to the syntax diagram::
> 
>     +-------+--------------------------+
>     | ".. " | directive type "::" data |
>     +-------+ directive block          |
>             |                          |
>             +--------------------------+
> 
> Looking at the definition of a directive, and descriptions of the
> existing directives, and your code, I've realized that there are rules
> we can apply here.  The physical directive data and block comprise up
> to 3 *logical* parts:
> 
> 1. Directive arguments.
> 2. Directive attributes (or "options", in command-line terms).
> 3. Directive content.

This is beginning to look good.

> Individual directives can use any combination of these:
> 
> - The admonitions directives ("note" etc.), "topic", "line-block", and
>   "parsed-literal" use content only, no arguments or attributes. (3)
> - "meta" also has only content; the content looks like an attribute
>   block, but it's parsed as a field list. (3)
> - "image" and "contents" each have one argument, and attributes. (1,2)
> - "figure" has one argument, attributes, and content. (1,2,3)
> - "sectnum" has attributes only. (2)
> - "target-notes" has nothing. ()
> 
> If a directive has arguments and/or attributes, they are taken from
> the directive data and directive block, up to the first blank line.
> (Note that the docs for the "raw" directive didn't have a blank line;
> that was a mistake, and I've fixed it.)  If the directive has content
> also, it is taken after the arguments and/or attributes, from the
> directive block.  If the directive has content only, it is taken from
> the directive data and directive block, up to the end of the
> indentation.
> 
> I think the "cheapDirective" function should be reworked to allow any
> variation of directive structure to be parsed, renamed to
> "parse_directive", and put into directives/__init__.py.  Let me know
> if you'd like to give this a try, or if you'd rather I did it.  But
> read on; there's a wrinkle ahead.

I think I can do that.

> > In order to give some context to the debate over :raw: vs.
> > :include:, I implemented both.
> 
> I think the "include" directive should be reStructuredText-only, and
> the "raw" directive should have an optional second argument, a path.
> They definitely should not duplicate each other's functionality
> (TOOWTDI).  In the terms defined above,

I'm not disagreeing, but I got a bit lost in the debate over :raw: vs.
:include: and wanted a) some context to help me form an opinion, and b)
to test the generalized directive parsing by using it for more than one
directive.

> - "include" should have one argument, a path. (1)
> 
> - "raw" should have one or two arguments -- a format ("html" etc.) and
>   an optional path -- and content, but only if there was no path
>   argument.  So this would be either a (1) or a (1,3) directive.  So
>   that forces us to split the parse in two, perhaps into
>   "parse_directive" (which parses the arguments & attributes), plus
>   "get_directive_content".

Not necessarily.  Is there any reason we can't parse a directive and
return a 3-tuple (arguments, options, content).  It would then be up to
the individual directive to test that these do have values, if required.

> Note that there are no attributes/options now.  If an attribute is
> required, it shouldn't be an attribute.  It's analogous to
> command-line options: "required option" is an oxymoron.  I'm thinking
> of changing the terminology in the spec from "attribute" to "option"
> to help reinforce this.

OK, but this means sub-parsing the arguments instead of pulling data out
of the attributes.  My preference would be to have a standard way to
specify not only the types of attributes, but default values for them. 
Then there are no optional attributes, just default or explicitly set. 
This is a more XML-ish way to go.

> Looking at the directives in the test1.rst file in order, first
> there's::
> 
>     .. include:: file:test2.rst
> 
> I'm not comfortable with URL syntax here.  I really think it's YAGNI,
> and may open up a big can of worms.  So that one should become::

Well, I think we *do* need URIs if we want includes to be useful to
ZReST (which is not file-system based, but lives in an object-oriented
database accessible by URI).  Obviously, we don't need a file: URI for
such a trivial (and file-based) example, that's just a way to test that
URIs work in general.

>     .. include:: test2.rst
> 
> The next one is an "include"/"raw" hybrid::
> 
>     .. include:: test3.rst
>        :raw: 
>        :format: html

Well, there were a few possibilities tossed around in email.  I couldn't
remember why the format was even needed (it's *raw*, what else do we
need?), but then my data wouldn't show up using html.py and I had to dig
through the writer code to figure out that raw nodes are tested for
format type.

> It should be::
> 
>     .. raw:: html test3.rst

OK, if that's the correct way to do it in reST, I'll do that.  I still
prefer the default attributes method discussed above rather than
multiple arguments because it makes the names and types of the
argument/option/attribute explicit.

> I noticed that you've got the attributes *after* the content in the
> next ones::
> 
>     .. include::
>        This is a <super>Test</super> of the <sub>Emergency</sub>
>        <strike>Broadcasting</strike> &quot;System&quot;
>        :raw:
>        :format: html

Yes, that was my mis-parsing of the directive (because I was working
from images.py, not the spec).  The way I was grabbing (or failing to
grab?) the content threw an exception if I put the attributes first, but
worked OK if I put them after.  I didn't like this either.

> Attributes always come *before* the content (who's to say that
> ":format: html" isn't valid raw data in some format?).  In any case,
> that directive should become::
> 
>     .. raw:: html

OK.  But the same arguments apply.  Who's to say :format: html isn't
valid raw data at the beginning of the data as well as easily as at the
end?  I agree that it's aethetically and functionally better to have the
attributes first, just not because the string ":format: html" might
appear in the data.

Also, the problem with using multiple comes up. Arguments to :raw: are 

path format
or
format

But it seems more intuitive to me to put the required argument first: so
the first argument to :raw: would always be format and we don't have to
run tests or special cases.  If there's a second argument, then it's
path.  This still isn't as clear and explicit (to me) as using
attributes, but better than having argument position be dependent on
number of arguments.

>        This is a <super>Test</super> of the <sub>Emergency</sub>
>        <strike>Broadcasting</strike> &quot;System&quot;
> 
> Next::
> 
>     .. raw::
>        This is <strong>RAW</strong>.  Really, <em>really,</em> raw.
>        :format: html
> 
> Should become::
> 
>     .. raw:: html
> 
>        This is <strong>RAW</strong>.  Really, <em>really,</em> raw.
> 
> (Note the blank line.)

Oh, there's a blank line *before* the content.  I didn't get that.  I
thought the directive *ends* with a blank line.  Yikes.  Now I see why
you want get_directive_content().  Doesn't that introduce all sorts of
possible ambiguities?  Shouldn't there be one format for directives in
all cases so that the users (typing in raw text with reST getting in
their way as little as possible) don't have to remember the special
cases for different directives?

> The last "raw" directive was::
> 
>     .. raw::
>        :include: test3.rst
>        :format: html
> 
> And should become::
> 
>     .. raw:: html test3.rst
> 
> (This is the same as the second directive, therefore redundant.)

The redundancy is deliberate.  I was testing two different possibilities
and wanted to be sure they both worked.  They do the same thing,
therefore they work.

> I've attached an edited test1.rst file.
> 
> > Let me know if anything should be done differently.  I've tried to
> > conform to the project policies (although reading through now I realize
> > that I used triple-single-quotes for docstrings rather than
> > triple-double-quotes, but the files are already zipped.  I'm cool with
> > criticism and still not terribly familiar with the internals, so any
> > advice on how to do it better would be appreciated.
> 
> I hope I haven't scared you off.  If you do decide to continue with
> this, it will give you a chance to fix the triple-single-quotes. ;-)
> Also, I noticed that you used non-standard 2-space indents in some
> places; naughty!

Ah yes, old habits die hard.

> BTW, I never knew that you could do sequence-unpacking on the
> exception arguments.  Thanks for teaching me something new!

My pleasure.  I learned it while reading the python docs on exception
handling last week while working on this, so in a way, you taught it to
me!

So what you'd like to see is:

:raw: [filepath] format

:include: filepath (which must be reST)

and functions for directives/__init__.py

parse_directive()

and 

get_directive_content()

right?

--Dethe

In the message "Re: [Docutils-develop] Re: Strange --stylesheet handling..."
    on Fri, 23 Aug 2002 21:39:28 -0400, David Goodger <go...@us...> wrote:

> I've redefined "--stylesheet" as containing an invariant URL, used
> verbatim.  Added "--stylesheet-path", interpreted w.r.t. the working
> directory.  Similarly for "--pep-stylesheet" and
> "--pep-stylesheet-path".

> The snapshot is ready for download.

This change works fine for me too. 
Thank you.

> -- 
> David Goodger  <go...@us...>  Open-source projects:
>   - Python Docutils: http://docutils.sourceforge.net/
>     (includes reStructuredText: http://docutils.sf.net/rst.html)
>   - The Go Tools Project: http://gotools.sourceforge.net/



> -------------------------------------------------------
> This sf.net email is sponsored by: OSDN - Tired of that same old
> cell phone?  Get a new here for FREE!
> https://www.inphonic.com/r.asp?r=sourceforge1&refcode1=vs3390
> _______________________________________________
> Docutils-develop mailing list
> Doc...@li...
> https://lists.sourceforge.net/lists/listinfo/docutils-develop

On Fri, 23 Aug 2002 21:39:28 -0400 David Goodger
<go...@us...> wrote:

> I've redefined "--stylesheet" as containing an invariant URL, used
> verbatim.  Added "--stylesheet-path", interpreted w.r.t. the working
> directory.  Similarly for "--pep-stylesheet" and
> "--pep-stylesheet-path".

Great. :)

---
Adam Chodorowski <ad...@ch...>

Yes, 9 mm submachine guns with slower firing rates would reduce weight 
problems, but it seems that real action heroes use Mac 10's, preferably one 
in each hand. We can't help asking where the sidekicks are with wheelbarrows 
to carry the ammo.
    -- http://intuitor.com/moviephysics/

On Fri, 23 Aug 2002 21:35:48 -0400 David Goodger
<go...@us...> wrote:

> > I see that the HTML writer creates the following elements that are
> > then joined in that order:
> > 
[...]
> Didn't I tell you about that?  Thought I did.  I went to the trouble
> of separating ``body_pre_docinfo`` and ``docinfo`` out of ``body``
> just for you!  

Yes, I know. Therefore I didn't ask about those elements since I knew what
they were about. I explicitly asked about...

> > I'm wondering exactly what head_prefix, head, body_prefix and
> > body_suffix actually contain...
> 
> Just run the code and see.  

Well... I guess I'll have to do that then. :)

---
Adam Chodorowski <ad...@ch...>

Do not meddle in the affairs of wizards, for you are crunchy and good
with ketchup.
    -- Aleister Crowley

There's been a lot of progress on "The Docutils Document Tree" lately.
See http://docutils.sf.net/spec/doctree.html.  There are 55 DTD
elements left to describe; I'd appreciate any help I can get.
Proofreading, editing, or writing; even one element section would be
great!

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

I've redefined "--stylesheet" as containing an invariant URL, used
verbatim.  Added "--stylesheet-path", interpreted w.r.t. the working
directory.  Similarly for "--pep-stylesheet" and
"--pep-stylesheet-path".

The snapshot is ready for download.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Adam Chodorowski wrote:
> I see that the HTML writer creates the following elements that are
> then joined in that order:
> 
> head_prefix
> head
> body_prefix
> body_pre_docinfo
> docinfo
> body
> body_suffix

Didn't I tell you about that?  Thought I did.  I went to the trouble
of separating ``body_pre_docinfo`` and ``docinfo`` out of ``body``
just for you!  (See the "Optionally disable docinfo section" thread.)
If you don't want the "docinfo", just concatenate everything in the
list above *except* ``docinfo``.  Use the ``io.NullOutput`` I/O class
to avoid unnecessary output.  You'll have to encode the text yourself.

> I'm wondering exactly what head_prefix, head, body_prefix and
> body_suffix actually contain...

Just run the code and see.  The parts listed above are just an
arbitrary division of the HTML output for convenience.  While
processing the document for HTML output, we may need to insert <meta>
tags inside <head>: put 'em in the ``head`` list.  The "footer"
element is actually located near the beginning of the document, so it
gets prepended to ``body_suffix``, before most of ``body`` is
finished.  And so on.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Hi.

I see that the HTML writer creates the following elements that are then joined
in that order: 

head_prefix
head
body_prefix
body_pre_docinfo
docinfo
body
body_suffix

I'm wondering exactly what head_prefix, head, body_prefix and body_suffix
actually contain...

---
Adam Chodorowski <ad...@ch...>

No matter how subtle the wizard, a knife in the shoulder blades will seriously
cramp his style.
    -- McCloctnik the Lucid

Kenichi Sato wrote:
> The following patch seems working for me.

Thanks; it's checked in (with modifications).  I still intend to fix the
command-line options somehow.  I currently like this option best:

  - Redefine "--stylesheet=URL" as an invariant, uninterpreted.  Add
    an interpreted "--stylesheet-path=PATH".  Similarly for
    "--pep-stylesheet".  The -path variant would override.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

In the message "Re: [Docutils-develop] Re: Strange --stylesheet handling..."
    on Wed, 21 Aug 2002 22:46:44 -0400, David Goodger <go...@us...> wrote:

> Kenichi Sato wrote:
> > I believe we can implement relative_uri() so that it returns
> > "default.css" for the case 1 and "../xyz/default.css" for the
> > case 4.
>
> If you'd like to give it a try, patches are very welcome.

Oops. Ok, I tried. The following patch seems working for me.

Index: utils.py
===================================================================
RCS file: /cvsroot/docutils/docutils/docutils/utils.py,v
retrieving revision 1.11
diff -c -r1.11 utils.py
*** utils.py	21 Aug 2002 02:43:46 -0000	1.11
--- utils.py	22 Aug 2002 12:04:24 -0000
***************
*** 364,370 ****
      """
      source_parts = os.path.abspath(source).split(os.sep)
      target_parts = os.path.abspath(target).split(os.sep)
!     if source_parts[:2] != target_parts[:2]:
          # Nothing in common between paths.  Return absolute path.
          return '/'.join(target_parts)
      source_parts.reverse()
--- 364,370 ----
      """
      source_parts = os.path.abspath(source).split(os.sep)
      target_parts = os.path.abspath(target).split(os.sep)
!     if source_parts[0] != target_parts[0]:
          # Nothing in common between paths.  Return absolute path.
          return '/'.join(target_parts)
      source_parts.reverse()


> > Wow, it's in Japanese.
>
> I lived in Gunma for 3 years and Tokyo for 4 years (Minato-ku,
> Shibaura).  My wife is Japanese, so I get to practise.

I see. 

-- 
Kenichi SATO

Thanks for the detailed explanation.

Kenichi Sato wrote:
> I believe we can implement relative_uri() so that it returns
> "default.css" for the case 1 and "../xyz/default.css" for the
> case 4.

If you'd like to give it a try, patches are very welcome.

> Wow, it's in Japanese.

I lived in Gunma for 3 years and Tokyo for 4 years (Minato-ku,
Shibaura).  My wife is Japanese, so I get to practise.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Hi David,

Thank you for the quick reply.

In the message "Re: [Docutils-develop] Re: Strange --stylesheet handling..."
    on Wed, 21 Aug 2002 18:59:38 -0400, David Goodger <go...@us...> wrote:

> You must be writing the HTML to a different disk than the one
> containing the source file or stylesheet.  Is this so?  I hadn't
> thought about that case.

No, I put the source, the stylesheet and the target in the same
disk and even in the same directory.

> If that's not the case, please show us the command you use (and any
> config files) which result in ``href="z:/default.css"``.

Ok, I should have explained in more details. I attached (mount) a
subdirectory of network drive as a virtual drive. 

For example, suppose I have a directory /tmp/docutils/tools on a
samba server in which "test.txt" and "default.css" are there. Now
I can mount the tools directory to Z: driver. Then the files are
seen as Z:\test.txt and Z:\default.css respectively.

Now, on Windows.

 bash> cd Z:
 bash> html.py test.txt test.html

Then, the stylesheet specification becomes href="z:/default.css".

REASON:

I looked at the source code and found that 

 * os.getcwd() returns the absolute path with the driver
   letter (e.g. "z:\\default.css")

 * docutils.utils.relative_uri() returns the absolute path if the
   common part of source and target is only the root directory.
   Therefore, 

     1.	relative_uri("z:\\test.html". "z:\\default.css") 
		=> "z:/default.css"

     2.	relative_uri("z:\\abc\\test.html". "z:\\abc\\default.css") 
		=> "default.css"

     3.	relative_uri("z:\\abc\\efg\\test.html". "z:\\abc\\hij\\default.css") 
		=> "../hij/default.css"

     4.	relative_uri("z:\\abc\\test.html". "z:\\xyz\\default.css") 
		=> "z:/xyz/default.css"

   I believe we can implement relative_uri() so that it returns
   "default.css" for the case 1 and "../xyz/default.css" for the
   case 4.

> > I understand the benefit of modifying the pathname to relative
> > one, but it would be better if we had an option to turn it off.

> I agree.  I'm not certain of the best approach as yet.  Here are the
> latest notes:

>   Fix the absolute/relative path handling.

>   All references to files (stylesheet, source file [in footer], PEP
>   template) are interpreted relative to the current working directory,
>   and adjusted relative to the destination directory.  For example::

>       cd /work
>       html.py -s --stylesheet=default.css /src/test.txt /dst/text.html

>   This will interpret the stylesheet as "/work/default.css", so the
>   stylesheet link in the HTML (which is written to the /dst dir) will
>   read "../work/default.css", and the source link (-s option) will
>   read "../src/test.txt".

>   - Perhaps the current behavior should be optional
>     ("--relative-paths" or something).  The default could be not to
>     process paths at all, just use them verbatim.
>
>   - We could add an "--verbatim-paths" option, which indicates that
>     paths should not be modified at all.

I didn't think about the other paths/URLs, but I would support the
option 1 at least for stylesheet. I mean, by default, the string
specified with --stylesheet option will appear in the <link
rel="stylesheet" ... line. Then, if you specify --relative-paths
(or --relative-stylesheet) option, the path is handled by
utils.relative_uri() method. 

(Or, my problem will simply be solved by just fixing the above
 relative_uri() glitch)

>   - Use variant options, such as "--stylesheet-url=URL", which
>     wouldn't be interpreted as paths.  Or leave "--stylesheet=URL"
>     uninterpreted, and add an interpreted "--stylesheet-path=PATH".

This option might be confusing (I can imagine a post will come to
the ML to ask the differences of these two :-)

>   Paths and URLs should be treated differently.  Sometimes a relative
>   path can be treated as the equivalent of a relative URL, but that's
>   not the case for absolute paths/URLs.  The "--pep-template" value is
>   *not* a URL, but a local file.  "--stylesheet" could be either.
>   Source links (generated in the footer) may depend on how the HTML is
>   installed.  I've been putting .html files in the same directories as
>   the source .txt files, but there are other ways to do it, and the
>   directory layout used at runtime may be different from the installed
>   layout.

> I'd like to come up with a consistent, clean solution.  Comments are
> welcome.

> 佐藤さん、フィードバックをいただいて、どうもありがとうございました。

Wow, it's in Japanese.

いいえ、どういたしまして。

It's my pleasure.

-- 
Kenichi SATO

Kenichi Sato wrote:
> I have the same problem and it is even worth for me because the
> driver letter (yes, I'm trying on Windows!) is put in front of the
> stylesheet path when it becomes the absolute one
> (e.g.  href="z:/default.css").  This is not what I expected and
> totally useless if I want to publish the html pages through a web
> server.  In my case, the trick found by Adam (specifying "../..."
> path) doesn't help either.

You must be writing the HTML to a different disk than the one
containing the source file or stylesheet.  Is this so?  I hadn't
thought about that case.

If that's not the case, please show us the command you use (and any
config files) which result in ``href="z:/default.css"``.

> I understand the benefit of modifying the pathname to relative
> one, but it would be better if we had an option to turn it off.

I agree.  I'm not certain of the best approach as yet.  Here are the
latest notes:

  Fix the absolute/relative path handling.

  All references to files (stylesheet, source file [in footer], PEP
  template) are interpreted relative to the current working directory,
  and adjusted relative to the destination directory.  For example::

      cd /work
      html.py -s --stylesheet=default.css /src/test.txt /dst/text.html

  This will interpret the stylesheet as "/work/default.css", so the
  stylesheet link in the HTML (which is written to the /dst dir) will
  read "../work/default.css", and the source link (-s option) will
  read "../src/test.txt".

  - Perhaps the current behavior should be optional
    ("--relative-paths" or something).  The default could be not to
    process paths at all, just use them verbatim.

  - We could add an "--verbatim-paths" option, which indicates that
    paths should not be modified at all.

  - Use variant options, such as "--stylesheet-url=URL", which
    wouldn't be interpreted as paths.  Or leave "--stylesheet=URL"
    uninterpreted, and add an interpreted "--stylesheet-path=PATH".

  Paths and URLs should be treated differently.  Sometimes a relative
  path can be treated as the equivalent of a relative URL, but that's
  not the case for absolute paths/URLs.  The "--pep-template" value is
  *not* a URL, but a local file.  "--stylesheet" could be either.
  Source links (generated in the footer) may depend on how the HTML is
  installed.  I've been putting .html files in the same directories as
  the source .txt files, but there are other ways to do it, and the
  directory layout used at runtime may be different from the installed
  layout.

I'd like to come up with a consistent, clean solution.  Comments are
welcome.

佐藤さん、フィードバックをいただいて、どうもありがとうございました。

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Hi,

> FROM: Adam Chodorowski
> DATE: 08/14/2002 15:21:13
> SUBJECT:  [Docutils-develop] Strange --stylesheet handling...
> 
> Hi.
> 
> I'm experiencing some strange handling of the --stylesheet option of the HTML
> writer, which I use like the following::
> 	
> 	stylesheet=aros.css
> 
> But the generated HTML contains::
> 
> 	<link rel="stylesheet" href="../../docs/aros.css" type="text/css" />
> 
> And that is not at all what I specified! It should be noted that the source
> file is in a different directory from the destination file, and that the
> current directory is the source one. I think this is the problem: for some
> reason the --stylesheet option is not taken verbatim, but is looked up
> relative to the current directory and "corrected". 

I have the same problem and it is even worth for me because the
driver letter (yes, I'm trying on Windows!) is put in front of the
stylesheet path when it becomes the absolute one
(e.g.  href="z:/default.css").  This is not what I expected and
totally useless if I want to publish the html pages through a web
server.  In my case, the trick found by Adam (specifying "../..."
path) doesn't help either.

I understand the benefit of modifying the pathname to relative
one, but it would be better if we had an option to turn it off.

Regards,

> This is totally useless for me, as I have separate build and source
> directories. If this is a intended feature (I do see the usefullness of it
> when building in a single directory), perhaps another option would be
> apropriate. Something like --absolute-stylesheet perhaps?
> 
> ---
> Adam Chodorowski <<EMAIL: PROTECTED>>
> 
> If studies were interesting, the world would come to a halt and nothing 
> would ever get done :) 
>     -- James Kehl
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by: Dice - The leading online job board
> for high-tech professionals. Search and apply for tech jobs today!
> http://seeker.dice.com/seeker.epl?rel_code=31
> _______________________________________________
> Docutils-develop mailing list
> <EMAIL: PROTECTED>
> https://lists.sourceforge.net/lists/listinfo/docutils-develop

-- 
Kenichi SATO

Attached is the announcement I sent to the Zope announcements mailing list.


    Richard

Garth Kidd wrote:
> So, does raw also work for raw XML to insert into XML output, raw PDF to
> insert into PDF output, raw PostScript to insert into PostScript output,
> and so on,

Yes.

> or are we assuming that it's only really HTML output that matters?

Never!

> Getting back to the implicit ``text/``, do the MIME content
> types for all of the suspect reST output types all begin with ``text/``?

I don't know about PDF, but I suspect this is a red herring.  I don't think
we have a big enough set of formats to justify using MIME types.  We've
already got a set of identifiers that are perfectly suitable: the Writer
format names.

OTOH, MIME types *are* future-proof.  (This was the reasoning behind the PEP
decision.)  But they feel like over-engineering, a classic case of "don't
add functionality before it's needed".  I'd rather wait until significant
use cases present themselves, if ever.

> I'd say that raw should have a ``:source:`` attribute. To me,
> ``include::`` would be used to pull in reST content from another
> document. 

Noted.  And I tend to agree.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

> I didn't mention that the "raw" directive's main function is::
> 
>     .. raw:: html
> 
>        <p>Arbitrary raw HTML here, will be inserted as-is
>        into HTML output.</p>

Oh, okay. Whups. 

So, does raw also work for raw XML to insert into XML output, raw PDF to
insert into PDF output, raw PostScript to insert into PostScript output,
and so on, or are we assuming that it's only really HTML output that
matters? Getting back to the implicit ``text/``, do the MIME content
types for all of the suspect reST output types all begin with ``text/``?


> Normally, the "raw" directive's content will be supplied 
> within the document body.  So the question becomes, does the 
> "raw" directive grow an "include" attribute (or optional 
> argument), or does the "include" directive grow a "raw" 
> attribute?  I'm not sure that "both" is a good answer here.

Aah, so that's the distinction. My apologies; I've been tuned out for a
little while, and was too lazy to read back on this thread. Oops. 

I'd say that raw should have a ``:source:`` attribute. To me,
``include::`` would be used to pull in reST content from another
document. 

> > Now, anyone want to explain why it's ``html`` and not a full MIME 
> > type? :) 
> 
> The "text/" part is implicit? ;-)  

That's enough for me. 

> But seriously, MIME types 
> *have* been adopted for reStructuredText-format PEPs 

Cool! 

Regards,
Garth.

Garth Kidd wrote:

>> So, should it be::
>> 
>>   .. raw:: html fragment.html
>> 
>> or::
>> 
>>   .. include:: fragment.html
>>      :raw: html
> 
> +1 to having both both available (i.e. ``raw::`` mapping directly to
> ``include::`` with ``:raw:`` set, or vice versa).

I didn't mention that the "raw" directive's main function is::

    .. raw:: html

       <p>Arbitrary raw HTML here, will be inserted as-is
       into HTML output.</p>

Normally, the "raw" directive's content will be supplied within the document
body.  So the question becomes, does the "raw" directive grow an "include"
attribute (or optional argument), or does the "include" directive grow a
"raw" attribute?  I'm not sure that "both" is a good answer here.

> Now, anyone want to explain why it's ``html`` and not a full MIME type?
> :) 

The "text/" part is implicit? ;-)  But seriously, MIME types *have* been
adopted for reStructuredText-format PEPs (official announcement any day
now), with "text/plain" as the default and "text/x-rst" as the new
alternative.  As for using MIME types with the "raw" directive here, I guess
I'd have to see how they'd be useful.  In other words, would such a MIME
type ever begin with anything *other* than "text/"?

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

> So, should it be::
> 
>     .. raw:: html fragment.html
> 
> or::
> 
>     .. include:: fragment.html
>        :raw: html

+1 to having both both available (i.e. ``raw::`` mapping directly to
``include::`` with ``:raw:`` set, or vice versa). 

Now, anyone want to explain why it's ``html`` and not a full MIME type?
:) 

Regards,
Garth.

On Fri, 2002-08-16 at 17:30, David Goodger wrote:
> Dethe Elza wrote:
> > I think I can handle creating a directive and even writing up how to
> > create a directive, but I'm not up for a revamp of the core system.
> 
> Those two would be great!  I'll be happy to take care of revamping
> ``utils.Reporter``.

Excellent.

> > Is image still the best directive to get started with?
> 
> Yes, it does all the parsing you'll need.  And also look at the
> directives/admonitions.py module to see how to use
> ``state.nested_parse()``.  If the files you're including contain
> section titles (which the general case would require), you'll need to
> pass in ``match_titles=1``.

Ah, that's the "recursive parse" I was thinking of!  Thanks.

--Dethe

I was updating the to-do list from the recent discussion, when I got to
thinking about the ":raw:" parameter:

    A "raw" flag (attribute with no value) would inform us that the
    imported data is raw (already in native format, like HTML).
    Perhaps there should be an attribute value, saying what the native
    format is.  Instead, perhaps the "raw" directive should take an
    "include" parameter (explicit attribute or second parameter in its
    directive data).

So, should it be::

    .. raw:: html fragment.html

or::

    .. include:: fragment.html
       :raw: html

?

A description of the "raw" directive is at
http://docutils.sf.net/spec/rst/directives.html#raw-data-pass-through.
The latest notes about the "include" directive are at
http://docutils.sf.net/spec/notes.html#misc-include.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Dethe Elza wrote:
>>> There could be another flag on the include directive, as there is
>>> in the recursive parse within reST I believe, telling it whether
>>> to process the included text within the same context or not, but I
>>> think the default should be to assume the same context.
>> 
>> What "recursive parse within reST" are you talking about?
> 
> Ummm.  There was something somewhere in the code.  I can't remember
> exactly where and I don't have a chance to dig it out right now.  I
> could be wrong.  Your mileage may vary.  Void where prohibited.

There *is* plenty of recursive parsing going on.  I just wasn't sure
which one you meant.  Not to worry.

>> Factoring in the above, we get::
>> 
>>     .. include:: path/subdocument.rst
>>        :raw: (true if set)
> 
> Looks good.
> 
>> That's feasible.  I wonder how to actually do the parse, though.  I
>> don't think it's a good idea to alter the already-read data, so a
>> separate, nested parse would probably be in order.  Also, the error
>> reporting mechanism would have to be revamped to include the file
>> which is the source of system messages.
> 
> Um.  I'm not sure if I'm up to revamping the error reporting
> mechanism. I'm trying to find something that's *easier* than
> 
>   cat file1.rst file2.rst file3.rst | html.py> finished.html
> 
> I think I can handle creating a directive and even writing up how to
> create a directive, but I'm not up for a revamp of the core system.

Those two would be great!  I'll be happy to take care of revamping
``utils.Reporter``.

> Is image still the best directive to get started with?

Yes, it does all the parsing you'll need.  And also look at the
directives/admonitions.py module to see how to use
``state.nested_parse()``.  If the files you're including contain
section titles (which the general case would require), you'll need to
pass in ``match_titles=1``.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/