fileno(stdin) versus STDIN_FILENO and friends

[Peter Dufault]

Shouldn't STDOUT_FILENO etc always be valid, regardless of thread?

Consider this code snippet installed in the RTEMS shell as "filenos":
===
static int
filenos_cmd(int argc, char **argv)
{
int stdin_fileno = fileno(stdin),
stdout_fileno = fileno(stdout),
stderr_fileno = fileno(stderr);

fprintf(stderr,
"fileno(stdin), STDIN_FILENO: %d %d\n"
"fileno(stdout), STDOUT_FILENO: %d %d\n"
"fileno(stderr), STDERR_FILENO: %d %d\n",
stdin_fileno, STDIN_FILENO,
stdout_fileno, STDOUT_FILENO,
stderr_fileno, STDERR_FILENO);

fprintf(stderr, "- %s\n",
(stdin_fileno == STDIN_FILENO &&
stdout_fileno == STDOUT_FILENO &&
stderr_fileno == STDERR_FILENO) ? "OK" : "BAD");

return 0;
}
====

Called from the console:

[/] # filenos
fileno(stdin), STDIN_FILENO: 0 0
fileno(stdout), STDOUT_FILENO: 1 1
fileno(stderr), STDERR_FILENO: 2 2
- OK
[/] #

Called via telnet:

[/] # filenos
fileno(stdin), STDIN_FILENO: 7 0
fileno(stdout), STDOUT_FILENO: 8 1
fileno(stderr), STDERR_FILENO: 9 2
- BAD
[/] #

Shouldn't the macros reference something that changes with the thread context to appropriate file descriptors without forcing reference to <stdio.h>?

Peter
-----------------
Peter Dufault
HD Associates, Inc. Software and System Engineering

[Eric Norum]

The document at http://pubs.opengroup.org/onlinepubs/009695399/basedefs/unistd.h.html implies that they really are supposed to be constants:

The following symbolic constants shall be defined for file streams:
STDERR_FILENO
File number of stderr; 2.
STDIN_FILENO
File number of stdin; 0.
STDOUT_FILENO
File number of stdout; 1.

but I agree that that is rather counterintuitive.

Post by Peter Dufault
Shouldn't STDOUT_FILENO etc always be valid, regardless of thread?
===
static int
filenos_cmd(int argc, char **argv)
{
int stdin_fileno = fileno(stdin),
stdout_fileno = fileno(stdout),
stderr_fileno = fileno(stderr);
fprintf(stderr,
"fileno(stdin), STDIN_FILENO: %d %d\n"
"fileno(stdout), STDOUT_FILENO: %d %d\n"
"fileno(stderr), STDERR_FILENO: %d %d\n",
stdin_fileno, STDIN_FILENO,
stdout_fileno, STDOUT_FILENO,
stderr_fileno, STDERR_FILENO);
fprintf(stderr, "- %s\n",
(stdin_fileno == STDIN_FILENO &&
stdout_fileno == STDOUT_FILENO &&
stderr_fileno == STDERR_FILENO) ? "OK" : "BAD");
return 0;
}
====
[/] # filenos
fileno(stdin), STDIN_FILENO: 0 0
fileno(stdout), STDOUT_FILENO: 1 1
fileno(stderr), STDERR_FILENO: 2 2
- OK
[/] #
[/] # filenos
fileno(stdin), STDIN_FILENO: 7 0
fileno(stdout), STDOUT_FILENO: 8 1
fileno(stderr), STDERR_FILENO: 9 2
- BAD
[/] #
Shouldn't the macros reference something that changes with the thread context to appropriate file descriptors without forcing reference to <stdio.h>?
Peter
-----------------
Peter Dufault
HD Associates, Inc. Software and System Engineering
_______________________________________________
rtems-users mailing list
http://www.rtems.org/mailman/listinfo/rtems-users

[Peter Dufault]

Post by Peter Dufault
STDERR_FILENO
File number of stderr; 2.
STDIN_FILENO
File number of stdin; 0.
STDOUT_FILENO
File number of stdout; 1.
but I agree that that is rather counterintuitive.

To me that means the streams should either be all shared across all threads or switched such that different threads have different stdin, stdout, stderr but are still files 0, 1, and 2.


Peter
-----------------
Peter Dufault
HD Associates, Inc. Software and System Engineering

[Peter Dufault]

Post by Peter Dufault
STDERR_FILENO
File number of stderr; 2.
STDIN_FILENO
File number of stdin; 0.
STDOUT_FILENO
File number of stdout; 1.
but I agree that that is rather counterintuitive.

Am I correct that threads are started with the streams stdin, stdout, and stderr set to files 0, 1, and 2, and don't inherit the file descriptors that the parent thread may have redirected them to? Is that dictated by any standard?

I think it would make more sense if stdin, stdout, and stderr are always files 0, 1, and 2, that the underlying files are switched at context switch time, and that threads inherit whatever underlying files the creating thread may have dup'd them to. Next in line would be to inherit the parent file descriptors.

Does anyone know about what the standards say about:

- stdin, stdout, stderr,
- file descriptors 0, 1, 2,
- STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO

and how this should play with pthreads? I haven't had much luck figuring it out.

Eric's earlier pointed-out-reference that STDIN_FILENO is by definition 0 implies to me that the way things are is at least confusing.


Peter
-----------------
Peter Dufault
HD Associates, Inc. Software and System Engineering

[Ralf Corsepius]

Post by Peter Dufault

Post by Peter Dufault
STDERR_FILENO
File number of stderr; 2.
STDIN_FILENO
File number of stdin; 0.
STDOUT_FILENO
File number of stdout; 1.
but I agree that that is rather counterintuitive.

Am I correct that threads are started with the streams stdin, stdout, and stderr set to files 0, 1, and 2,

yes.

Post by Peter Dufault
and don't inherit the file descriptors that the parent thread may have redirected them to?

The are global FILE*'s which are supposed to default to fileno's
0, 1, 2.

Post by Peter Dufault
Is that dictated by any standard?

see the link below.

Post by Peter Dufault
I think it would make more sense if stdin, stdout, and stderr are always files 0, 1, and 2,

This is not safe to assume. stdin, stdout and stderr are FILE*'s and can
be redirected, repopened or even be closed.

Post by Peter Dufault
that the underlying files are switched at context switch time, and that threads inherit whatever underlying files the creating thread may have dup'd them to. Next in line would be to inherit the parent file descriptors.
- stdin, stdout, stderr,
- file descriptors 0, 1, 2,
- STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO
and how this should play with pthreads?

FILE* are per-process and not per-thread.

Post by Peter Dufault
I haven't had much luck figuring it out.

The best I could find is this:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/stderr.html

Ralf

[Peter Dufault]

Post by Ralf Corsepius

Post by Peter Dufault
I think it would make more sense if stdin, stdout, and stderr are always files 0, 1, and 2,

This is not safe to assume. stdin, stdout and stderr are FILE*'s and can
be redirected, repopened or even be closed.

You are correct. Note that the reference you provided says about STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO (emphasis of "**" added):

"[CX]

[Chris Johns]

static void *hello(void *arg)
{
fprintf(stderr, "Hello stderr!\n");
return (void *)0;
}
static int
hello_stderr(int argc, char **argv)
{
pthread_t t;
pthread_create(&t, 0, hello, 0);
return 0;
}
[/] # filenos
fileno(stdin), STDIN_FILENO: 0 0
fileno(stdout), STDOUT_FILENO: 1 1
fileno(stderr), STDERR_FILENO: 2 2
[/] # hello
[/] # Hello stderr!
[/] # filenos
fileno(stdin), STDIN_FILENO: 7 0
fileno(stdout), STDOUT_FILENO: 8 1
fileno(stderr), STDERR_FILENO: 9 2
[/] # hello
[/] #
syslog: telnetd: accepted connection from 192.168.240.6 on /dev/pty0
Hello stderr!

This does not seem to follow the intended purpose of telnet for RTEMS.

So, stdin, stdout, and stderr are redirected per-thread, but descendants of the threads revert to 0, 1, and 2.
Is this in described by any standard?

I suspect not.

If not, and we're just dealing with undefined behavior, is there any reason not to make it optional that descendants of a thread inherit the file descriptors for stdin, stdout, and stderr from the parent thread? I'd find that behavior a lot more convenient, for example, threads started from telnet where the stderr went to the telnet window would still go to the telnet window. I consider it important to route stdout and stderr to reasonable places, for example, an error log in a window some place, and I'm trying to setup an environment for less experienced users where things work well.

I think it is reasonable to make this change. The whole shell and telnet
on RTEMS is an application trick to make the RTEMS "process" appear
multi-user and process based. We should attempt to make this emulation
appear as standard.

Chris

[Peter Dufault]

Post by Chris Johns
I think it is reasonable to make this change. The whole shell and telnet
on RTEMS is an application trick to make the RTEMS "process" appear
multi-user and process based. We should attempt to make this emulation
appear as standard.

I think (but don't know, but would like to know) that we're talking about behavior that isn't defined by a standard. If the behavior is defined, then RTEMS should ensure things follow the standard, but I haven't located a definition yet.

If we're operating outside of any definition then I suggest RTEMS make the stdio thread inheritance behavior optional: I'd prefer an inherited stdin,stdout,stderr, and I'll look at what it takes to make it work in the newlib framework. Others might like the way it works now, where all threads revert to file descriptors 0, 1, and 2 for stdin, stdout, and stderr for newly created threads.

Peter
-----------------
Peter Dufault
HD Associates, Inc. Software and System Engineering

[Chris Johns]

Post by Peter Dufault

Post by Chris Johns
I think it is reasonable to make this change. The whole shell and telnet
on RTEMS is an application trick to make the RTEMS "process" appear
multi-user and process based. We should attempt to make this emulation
appear as standard.

I think (but don't know, but would like to know) that we're talking about behavior that isn't defined by a standard. If the behavior is defined, then RTEMS should ensure things follow the standard, but I haven't located a definition yet.

I do not know of a standard for managing these things inside a process
environment. The standards talk about processes and this is what we are
attempting to emulate with the shell and telnet. This is what I was
referring to.

Post by Peter Dufault
If we're operating outside of any definition

I think we are because you are using part of some application type code
that is the shell and telnet and here we would like the process
standards to apply if they make sense and it can be done.

Post by Peter Dufault
then I suggest RTEMS make the stdio thread inheritance behavior optional: I'd prefer an inherited stdin,stdout,stderr, and I'll look at what it takes to make it work in the newlib framework. Others might like the way it works now, where all threads revert to file descriptors 0, 1, and 2 for stdin, stdout, and stderr for newly created threads.

I agree for the shell and telnet type code and here I would make the
behaviour you want as the default.

For me the general thread create case will have wait and be based on
what you discover.

Chris

[Peter Dufault]

Post by Chris Johns

Post by Peter Dufault
I think (but don't know, but would like to know) that we're talking about behavior that isn't defined by a standard. If the behavior is defined, then RTEMS should ensure things follow the standard, but I haven't located a definition yet.

I do not know of a standard for managing these things inside a process
environment. The standards talk about processes and this is what we are
attempting to emulate with the shell and telnet. This is what I was
referring to.

In my opinion an RTEMS collection of POSIX threads is a single process. The behavior of a single RTEMS process should match what you'd see in a single process in a POSIX compliant system. That's why the fact that the file pointers stdin, stdout, and stderr are part of the thread context is convenient but surprising, and as long as we're offering surprising behavior I'd like to make it optionally different, i.e., optionally determine which file pointers are inherited at pthread_create() time inherited.

Post by Chris Johns

Post by Peter Dufault
If we're operating outside of any definition

I think we are because you are using part of some application type code
that is the shell and telnet and here we would like the process
standards to apply if they make sense and it can be done.

Post by Peter Dufault
then I suggest RTEMS make the stdio thread inheritance behavior optional: I'd prefer an inherited stdin,stdout,stderr, and I'll look at what it takes to make it work in the newlib framework. Others might like the way it works now, where all threads revert to file descriptors 0, 1, and 2 for stdin, stdout, and stderr for newly created threads.

I agree for the shell and telnet type code and here I would make the
behaviour you want as the default.

Since I'd like it to apply to all code (for example, pthread code brought in in a library) without modification I think this has to happen at the pthread attribute level and in a task creation hook called for all tasks. The interface could be a custom pthread attribute, the implementation could be a task creation hook that checked the setting and used thread specific data for the file pointers. Or something, I have to look to see how it's currently done.

Peter
-----------------
Peter Dufault
HD Associates, Inc. Software and System Engineering

[Ralf Corsepius]

Post by Peter Dufault

Post by Chris Johns

Post by Peter Dufault
I think (but don't know, but would like to know) that we're talking about behavior that isn't defined by a standard. If the behavior is defined, then RTEMS should ensure things follow the standard, but I haven't located a definition yet.

I do not know of a standard for managing these things inside a process
environment. The standards talk about processes and this is what we are
attempting to emulate with the shell and telnet. This is what I was
referring to.

In my opinion an RTEMS collection of POSIX threads is a single process.

This is right. RTEMS is a single process.

Ralf

[Gedare Bloom]

I think there is an interesting philosophical debate here. Telnet and
shell are logically separate from an application, but RTEMS is only
designed to provide a single 'process'. So according to the principle
of least surprise, anything that is per-process should be shared by
all threads. Ralf pointed to a source that says these FILE*
'constants' should be per-process, so it is reasonable for a user to
expect them to be shared by all threads. Since telnet and shell are
emulating separate processes, however, the abstraction (and RTEMS'
threading model) breaks.

Since treating telnet/shell as separate processes is arguably a good
thing, support for process-like constructs seems useful. However, the
default behavior should be to treat all threads as part of a single
(logical) process unless otherwise managed, which means the per-thread
FILE* is wrong.

On Tue, Mar 15, 2011 at 4:51 AM, Ralf Corsepius

Post by Ralf Corsepius

Post by Peter Dufault

Post by Chris Johns

Post by Peter Dufault
I think (but don't know, but would like to know) that we're talking
about behavior that isn't defined by a standard. If the behavior is defined,
then RTEMS should ensure things follow the standard, but I haven't located a
definition yet.

I do not know of a standard for managing these things inside a process
environment. The standards talk about processes and this is what we are
attempting to emulate with the shell and telnet. This is what I was
referring to.

In my opinion an RTEMS collection of POSIX threads is a single process.

This is right. RTEMS is a single process.
Ralf
_______________________________________________
rtems-users mailing list
http://www.rtems.org/mailman/listinfo/rtems-users

[Till Straumann]

Post by Gedare Bloom
I think there is an interesting philosophical debate here. Telnet and
shell are logically separate from an application, but RTEMS is only
designed to provide a single 'process'. So according to the principle
of least surprise, anything that is per-process should be shared by
all threads. Ralf pointed to a source that says these FILE*
'constants' should be per-process, so it is reasonable for a user to
expect them to be shared by all threads. Since telnet and shell are
emulating separate processes, however, the abstraction (and RTEMS'
threading model) breaks.
Since treating telnet/shell as separate processes is arguably a good
thing, support for process-like constructs seems useful. However, the
default behavior should be to treat all threads as part of a single
(logical) process unless otherwise managed, which means the per-thread
FILE* is wrong.

I like your assessment. I believe the newlib semantics also influence
this discussion. IIRC they are as follows:

In general, FILE objects are 'global', i.e., a FILE opened by one
thread doesn't go away when the thread is deleted. However,
unless the patches provided with bugzilla #1247 etc. are applied
there are race conditions when multiple threads access the same FILE
(or try to fopen() different files concurrently).
(Sidenote: other newlib objects are also unprotected unless 1247 is
fixed).

HOWEVER: stdin/stdout/stderr are treated differently by newlib.
These three FILEs are 'per-thread' entities and are destroyed
(albeit this currently may still leak memory, see PR #1246)
when a thread is deleted.

Note that the stdin/stdout/stderr FILEs are 'lazy-initialized'
upon first use, i.e., a thread which doesn't use them doesn't
create them.

-> stdin/stdout/stderr refer to different streams/buffers
for each thread (but read/write the same underlying file
descriptors) and the buffers are 'attached' to the
thread's 'reent' struct.
-> other FILEs are global and attached to the global 'reent'
struct.


-- Till

Post by Gedare Bloom
On Tue, Mar 15, 2011 at 4:51 AM, Ralf Corsepius

Post by Ralf Corsepius

Post by Peter Dufault

Post by Chris Johns

Post by Peter Dufault
I think (but don't know, but would like to know) that we're talking
about behavior that isn't defined by a standard. If the behavior is defined,
then RTEMS should ensure things follow the standard, but I haven't located a
definition yet.

I do not know of a standard for managing these things inside a process
environment. The standards talk about processes and this is what we are
attempting to emulate with the shell and telnet. This is what I was
referring to.

In my opinion an RTEMS collection of POSIX threads is a single process.

This is right. RTEMS is a single process.
Ralf
_______________________________________________
rtems-users mailing list
http://www.rtems.org/mailman/listinfo/rtems-users

_______________________________________________
rtems-users mailing list
http://www.rtems.org/mailman/listinfo/rtems-users

[Peter Dufault]

Post by Till Straumann
In general, FILE objects are 'global', i.e., a FILE opened by one
thread doesn't go away when the thread is deleted. However,
unless the patches provided with bugzilla #1247 etc. are applied
there are race conditions when multiple threads access the same FILE
(or try to fopen() different files concurrently).
(Sidenote: other newlib objects are also unprotected unless 1247 is
fixed).
HOWEVER: stdin/stdout/stderr are treated differently by newlib.
These three FILEs are 'per-thread' entities and are destroyed
(albeit this currently may still leak memory, see PR #1246)
when a thread is deleted.
Note that the stdin/stdout/stderr FILEs are 'lazy-initialized'
upon first use, i.e., a thread which doesn't use them doesn't
create them.
-> stdin/stdout/stderr refer to different streams/buffers
for each thread (but read/write the same underlying file
descriptors) and the buffers are 'attached' to the
thread's 'reent' struct.
-> other FILEs are global and attached to the global 'reent'
struct.

I think the non-standard behavior I want is for file descriptors 0, 1, and 2 to be different for each thread and optionally inherited from the creating thread. I need to think about it some more and maybe try it out, I think it can be rigged up with a task-switch hook using dup2 as long as you're willing to keep a spare open file descriptor around. You could dup2 it out to descriptor at switch-out time and then back in at switch-in time.

Peter
-----------------
Peter Dufault
HD Associates, Inc. Software and System Engineering