CSC 357 Lecture Notes Week 4

CSC 357 Lecture Notes Week 4
Unbuffered File I/O UNIX Files and Directories

Relevant reading:
1. Stevens chapters 3 and 4.
2. Skim chapter 2.
Brief overview of C and UNIX standards (Stevens Chapter 2).
1. There are two levels of standards for the C language and UNIX library functions that support the language.
2. The ISO C standard defines the language proper, and the C standard library.
  1. Appendix A of K&R is the reference manual for the language proper.
  2. Appendix B of K&R is a summary of the major library components.
  3. The ISO (International Standards Organization) maintains the official standard, called ISO/IEC 9899:1999, the most recent update to which is November 2004.
3. The IEEE POSIX defines the full library standard for UNIX, UNIX-like, and other operating systems.
  1. The standard is based on UNIX, but any operating system may meet the standard.
  2. Systems that do are all POSIX compliant.
  3. POSIX includes the ISO standard C library, but not the specification of the language proper.
4. POSIX is a specification of library functions, not an implementation.
  1. There are many implementations of UNIX, many of which are POSIX compliant, or claim to be.
  2. IEEE has an official POSIX certification program to which UNIX implementors can apply and be certified for.
  3. The four implementations of UNIX that Stevens refers to are:
    1. Solaris from SUN Microsystems, used on falcon/hornet
    2. Linux, the widely-distributed open-source version of UNIX
    3. Mac OS X, from Apple
    4. FreeBSD, another open source version of UNIX
UNIX unbuffered file I/O (Stevens Chapter 3).
1. Most UNIX file I/O can be performed using only five functions -- open, read, write, lseek, and close.
2. These functions operate on file descriptors, at the UNIX kernel level.
3. They are lower-level functions than the "f" series, like fopen.
  1. These lower-level functions are referred to as unbuffered, since the operating system does not do any behind-the-scenes I/O data buffering.
  2. The OS does perform buffering on FILE* streams, but not with files accessed through lower level file descriptors.
  3. Section 5.4 of Stevens talks about buffering in detail
File descriptors (Stevens Section 3.2).
1. At the kernel level, all files are referred to by a file descriptor, which is a non-negative integer.
2. The open function returns a file descriptor.
3. Functions like read and write take file descriptors as inputs.

open (Stevens Section 3.3).

Open a file, returning its file descriptor, or -1 if error.
Signature:
int open(const char *pathname, intoflag, ... /* mode_t mode */);
1. pathname is the name of the file to open or create
2. oflag is used to specify options
3. the optional mode is only applicable when a new file is being created

Options values are constructed by a bitwise-inclusive-OR of flags from the following lists, defined in <fcntl.h>.

Applications must specify exactly one of the following three values for the file access mode:

O_RDONLY Open for reading only.
O_WRONLY Open for writing only.
O_RDWR Open for reading and writing.

Any combination of the following may be used:

O_APPEND Append to end of file on each write.
O_CREAT Create the file if it does not exist.
O_EXCL Fail O_CREAT if file exists.
O_TRUNC Truncate length to 0.
O_NOCTTY Do not have a terminal device become the controlling terminal.
O_NONBLOCK Do not block on open or for data to become available

POSIX synchronization options are:

O_DSYNC Wait for write to complete, but not for attribute updates.
O_RSYNC Have reads wait for pending writes.
O_SYNC Wait for write to complete, including for attribute updates.

There are other platform-specific options for such things as symbolic links, locks, and 64-bit file offsets.

The following example opens a file named "data" for reading and appended writing:

open("data", O_RDWR | O_APPEND)

creat (Stevens Section 3.4).
1. Create a file.
2. It's equivalent to the following call to open, which means that creat is effectively obsolete:
  open(pathname, O_WRONLY | O_CREAT | O_TRUNC, mode)
close (Stevens Section 3.5).
1. Close an open file, returning 0 if OK, -1 if error.
2. Signature:
  int close(int filedes);
3. When a process terminates, all open files are closed by the kernel.
lseek (Stevens Section 3.6).
1. The lseek function sets the read/write offset of an open file, returning new offset if OK, -1 if error.
  1. All open files have an offset position that defines from what byte a read starts or to what byte a write starts.
  2. The offset is initialized to 0 by open, unless O_APPEND is specified.
2. Signature:
  off_t lseek(int filedes, off_t offset, intwhence);
3. The interpretation of offset is based the value of whence, which must be one of the following, defined in <unistd.h>:
  - If whence is SEEK_SET, the offset is set to offset bytes from the beginning of the file.
  - If whence is SEEK_CUR, the offset is set to its current value plus offset; the offset value can be positive or negative.
  - If whence is SEEK_END, the offset is set to the size of the file plus offset.
4. The programmer can determine the value of the current offset without changing it by supplying an offset value of 0, e.g.,
```
off_t curpos;
curpos = lseek(fd, 0, SEEK_CUR);
```
  1. This can also be used to determine if a file is capable of seeking, which, e.g., stdin from a terminal is not.
  2. See the example on Page 64 of Stevens for further details on testing for the seekability of a file.
5. When lseek is used to set a file's offset larger than its current size, the file is said to have "a hole" in it.
  1. Operating systems may take advantage of this by allocating fewer file blocks for files with holes.
  2. Unwritten bytes in file holes read back as 0s, whether or not there is actual disk storage backing the hole.
  3. See the example on pp. 65-66 of Stevens for further details.
6. The type off_t allows the implementation of an OS to provide different size integers for file offsets, and hence the maximum size file that a program can work with.
  1. Most platforms these days support both 32-bit and 64-bit file offsets, the latter allowing files larger than 2 GB (2³¹-1 bytes).
  2. For example, here are the alternative definitions of off_t on falcon in /usr/include/stdio.h:
```
#if defined(_LP64) || _FILE_OFFSET_BITS == 32
typedef long off_t;
#else
typedef __longlong_t off_t;
#endif
```
read (Stevens Section 3.7).
1. Read from an open file, returning number of bytes read, 0 if eof, -1 if error
2. Signature:
  ssize_t read(int fildes, void *buf, size_tnbytes);
  1. The ssize_t return value is the number of bytes read, 0 on eof.
  2. The files is the file to read from, which must be open.
  3. The buf is the buffer of at least nbytes into which data are read.
3. There are several cases in which the number of bytes read is less than requested, including:
  1. If eof is reached during the read, the number of bytes read may be less than the number requested.
  2. When reading from a terminal device, normally only one line at a time is read.
  3. When reading from a network, buffering may cause fewer bytes than requested to be read.
  4. When reading from a pipe, only the number of available bytes is read.
  5. When reading from a record-oriented device, sometimes only a record at a time is read.
  6. When the read is interrupted by a signal, the read may only be partially completed.
4. The read operation starts at the current file offset.
5. After a successful or partially successful read, the file offset is incremented by the number of bytes actually read.
6. The typedefs ssize_t and size_t allow implementations flexibility in the number of bytes readable and requestable by one call to read.
write (Stevens Section 3.8).
1. Write data to an open file, returning number of bytes written if OK, -1 if error.
2. Signature:
  ssize_t write(int fildes, const void *buf, size_t nbytes);
3. The write starts at the current file offset of the given filedes, unless O_APPEND was set on open, in which case the file offset is set to the end of file before writing starts.
4. After a successful write, the file offset is incremented by the number of bytes actually written.
5. Typical causes for write failure are a full disk or exceeding the file size limit for a process; -1 is returned for these or other types of failure.
I/O Efficiency (Stevens Section 3.9).
1. This section of Stevens has some interesting data on the effect of the programmer-selected buffer size on execution time of read and write functions.
2. We'll discuss this issue further when we compare timing for unbuffered versus buffered I/O functions, in an upcoming lecture.
File sharing (Stevens Section 3.10).
1. Two or more processes ¹ can share the same file.
2. When they do, they have a common pointer to the same physical file data.
3. But the processes have independent copies of the following data:
  1. the file descriptor and its flags
  2. file status flags
  3. current file offset
4. The pictures on pp. 72 and 73 illustrate things well.
5. If file-sharing processes only read the file, there are no problems.
6. However, if processes each try to write to a shared file, they can potentially interfere with each other's work.
7. This is a classic "readers/writers" situation.
Atomic operations (Stevens Section 3.11).
1. The potential problem with multiple writers has to do with the fact that the operation sequence of lseek followed immediately by write is not guaranteed to be a single atomic operation.
  1. Specifically, a process can seek to a file location, but then be suspended before it gets a chance to do the write.
  2. If during the suspension another process does a seek and write, then the suspended process may end up writing to some unexpected place in the file.
2. For example, suppose processes A and B have a shared file.
  1. Process A does a seek to the end of the file and is then suspended by the kernel.
  2. Process B then does a seek to the end of the file, and then writes 100 bytes.
  3. Process A then gets reactivated to do its write, but the point it thinks is at the end of the file is now 100 bytes in front of the end.
3. To address this problem, there are the functions pwrite and pread, which perform an uninterruptible pair of operations -- seek followed by the write or the read.
4. The signatures are:
  ssize_t pwrite(int fildes, const void *buf, size_t nbytes, off_toffset);
  
  ssize_t pread(int fildes, void *buf, size_tnbytes, off_t offset);
5. There is also a potential problem with creating a file, when the intent is not to create it if it already exists. E.g.,
  1. Process A checks if a file exists, with the intent not to create it if it does.
  2. Process A is suspended, and process B gets control.
  3. Process B creates the file that A just checked, and it turns out the file did not exist before B creates it.
  4. Process A gets control back, thinks the file does not exist, and proceeds to re-create it, potentially interfering with what B did.
  5. E.g., there's a problem if B wrote to the file before A got control back, and then A re-creates it with the truncation option on.
6. In general, the term atomic operation refers to an operation that may be composed of multiple steps, but the steps are performed in an uninterruptible sequence.
  1. A subset of the steps cannot be performed.
  2. Either all steps run to completion, or none of them runs at all.
dup and dup2 (Stevens Section 3.12).
1. File descriptors can be duplicated using these functions.
2. The only difference between dup'd descriptors is the value of their file descriptor flags, of which there is only one defined by POSIX.
3. Duplicated descriptors share the same status flags, current offset, and file data.
4. We'll discuss the relevance of duplicated descriptors later, when we talk about the process exec function.
fsync
1. UNIX kernels typically use buffer caches to make read/write operations more efficient.
2. This means that the contents of cache memory and the contents of a file may differ until the cache is synchronized with the file.
3. For applications that care about this, e.g., database systems, the fsync function forces the synchronization of the cache and the associated file.
fcntl (Stevens Section 3.14).
1. The fcntl function provides for control of open files.
2. Signature:
  int fcntl(int fildes, int cmd, ... /*arg */ );
  1. The cmd is a #defined command value from <fcntl.h>.
  2. The optional arg varies based on the value of cmd.
3. There are a myriad of different cmds and args to control file properties.
4. Many of the properties that are settable with fcntl can be set when a file is opened; the fcntl function is useful because
  1. it allows file properties to be changed, without closing and reopening the file;
  2. in the case of stdio and pipes, fcntl is the only way to set file properties, when an application did not itself open the file associated with the stdio or pipe descriptor.
ioctl (Stevens Section 3.15).
1. The ioctl function provides control of file descriptors associated with devices.
2. Signature:
  int iocntl(int fildes, int request, ... );
  1. request and an optional third argument are interpreted by the device driver
  2. Generally the interpretation is performed in a very device-specific way.
/dev/fd
1. One of the nice things about UNIX is the uniform way that it treats files and devices.
  1. There is a standard directory named "/dev" that contains files associated with the hardware devices connected to a machine.
  2. We'll be seeing more about /dev files in upcoming lectures.
2. At the level of file descriptors, many UNIX systems provide a /dev/fd subdirectory, with files numbered 0, 1, 2, and possibly higher.
  1. By convention, file descriptors 0, 1, and 2 correspond to stdin, stdout, and stderr respectively.
  2. Having corresponding /dev/fd files for these enforces the uniformity of files and devices, and makes certain aspects of shell use cleaner.
3. The conventional association of stdio with the numeric file descriptors 0, 1, and 2 is not a POSIX thing.
  1. POSIX requires the definition of three symbolic constants STDIO_FILENO, STDOUT_FILENO, and STDERR_FILENO.
  2. Despite this POSIX generality, many UNIX applications, including shells, rely on the hard numeric mapping of file descriptors 0, 1, and 2.
Files and directories (Stevens Chapter 4).
1. A fundamental part of any operating system is the structure of its files and directories.
2. The UNIX structure treats files and directories pretty uniformly, in that a directory is itself a file.
3. And as noted above, UNIX treats files and devices in a reasonably uniform manner, though a device file is different in structural detail than a regular data file.
4. UNIX also provides the symbolic link file type, which is a file that points to another file.
5. At the level of system calls, there are stat functions that provide the information maintained by the OS for all files.
6. There are also other useful system functions that operate on files and directories.
stat, fstat, and lstat (Stevens Section 4.2).
1. These three functions return information about a file in a struct stat, which is defined in <sys/stat.h>.
2. Signatures:
  int stat(const char* restrict ²pathname, struct stat* restrict buf );
  
  int lstat(const char* restrict pathname, struct stat* restrictbuf );
  
  int fstat(int fildes, struct stat* buf );
  1. The returned data is in the buf parameter, which must point to a caller-declared structure.
  2. For fstat, the filedes parameter is the file descriptor of an open file.
  3. For each function, the return value is 0 if OK, -1 if error.
3. The difference between stat and lstat is that the latter returns the information about a symbolic link file, not the file referenced by the link; i.e., stat follows the symbolic link pointer, lstat does not.
4. Here's the definition of struct stat in use on falcon/hornet:
```
struct stat {
    dev_t       st_dev;                 /* device number */
    ino_t       st_ino;                 /* i-node number */
    mode_t      st_mode;                /* file type and mode */
    nlink_t     st_nlink;               /* number of links */
    uid_t       st_uid;                 /* user ID of owner */
    gid_t       st_gid;                 /* group ID of owner */
    dev_t       st_rdev;                /* device number for special files */
    off_t       st_size;                /* size in bytes, for regular files */
    timestruc_t st_atim;                /* time of last access */
    timestruc_t st_mtim;                /* time of last modification */
    timestruc_t st_ctim;                /* time of last file status change */
    blksize_t   st_blksize;             /* best I/O block size */
    blkcnt_t    st_blocks;              /* number of disk blocks alloc'd */
    char        st_fstype[_ST_FSTYPSZ]; /* type of file system */
};
```
  1. Note that all but the last struct field are declared as system-defined datatypes, which are defined in <sys/types.h> and elsewhere.
  2. Use of struct stat will figure prominently in your implementation of programming assignment 3.
File types (Stevens Section 4.3).
1. The two most commonly used types of file are regular data files and directories.
2. UNIX defines seven different files types:
  1. Regular file, which holds data of some kind; the UNIX kernel does not distinguish between text and binary data, leaving file data interpretation to the application programs that process a file.
  2. Directory file, which contains the names of other files and pointers to the file information.
  3. Block special file, which provides buffered I/O access to devices that communicate in fixed-sized blocks, such as disk drives.
  4. Character special file, which provides unbuffered I/O access to devices that communicate in variable-sized units.
  5. FIFO, which is a type of file for communication between processes, also called a named pipe.
  6. Socket, used for inter-process communication, including across a network.
  7. Symbolic link, which is a file that points to another file; symbolic links are akin to short cuts in Windows.
3. Page 90 of Stevens has a useful code example.
  1. It's a program that prints the file-type of each command-line argument.
  2. It uses lstat to obtain the file information.
4. Later in the chapter, on pages 121-125, there is another code example that uses lstat to traverse a directory hierarchy.

Footnotes:

¹ As defined in Chapter 1 of Stevens, a process is an independently executing program.

² As described on page 26 of Chapter 1 in Stevens, restrict is keyword
added to the 1999 ISO standard of C, and hence not covered in K&R. This
keyword is used to tell the compiler which function parameters can be optimized
within a function. Its use does not change the semantics of the function, only
potentially its performance. You need not use restrict in any of your
357 programs.

`O_RDONLY`	Open for reading only.
`O_WRONLY`	Open for writing only.
`O_RDWR`	Open for reading and writing.

O_APPEND	Append to end of file on each write.
O_CREAT	Create the file if it does not exist.
O_EXCL	Fail O_CREAT if file exists.
O_TRUNC	Truncate length to 0.
O_NOCTTY	Do not have a terminal device become the controlling terminal.
O_NONBLOCK	Do not block on open or for data to become available

O_DSYNC	Wait for write to complete, but not for attribute updates.
O_RSYNC	Have reads wait for pending writes.
O_SYNC	Wait for write to complete, including for attribute updates.