Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: files open in C (Read 5831 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

files open in C

Hi,

I tried out this code segment is VC 6


  for(i=0; i<1000; i++){
    stream = fopen("filename0xxx.txt", "w");
  }


where "filename0xxx.txt", xxx is different for different value of i..

My program will crash! It seemed that there is a upper limit to the number of files that a program can open this way.. Is there any way to increased the number of files that can be opened .. Maybe 10 000 files ? 

files open in C

Reply #1 –
Quote
http://msdn.microsoft.com/library/default....le_handling.asp
The C run-time libraries have a 512 limit for the number of files that can be open at any one time. Attempting to open more than the maximum number of file descriptors or file streams causes program failure. Use _setmaxstdio to change this number.


Using win32 API may be the answer...

Menno

files open in C

Reply #2 –
Quote
http://msdn.microsoft.com/library/default....setmaxstdio.asp
Note  This upper limit may be beyond what is supported by a particular Win32 platform and configuration.


Basically what this is saying is that having too many files open at the same time is not a good idea, it might break on some setups/platforms or Windows versions.

I am intrigued - what are you writing that could possibly require so many open files?

files open in C

Reply #3 –
Actually, for normal programming purposes, this limit will not be touch.. However, for R&D purposes, where "every gear change, every engine rev, the behaviour and performance of every component" has to be recorded and studied.. this limit will be exceeded very easily!

It is for designing somekind of "software telemetry".. for studying and debugging purposes..

files open in C

Reply #4 –
Yes, for C stdio there is definetely a limit, probably because of stdio implementation. K&R suggests that there is a static, fixed-size array of FILE objects, to which fopen returns pointers to. Undoubdly a great number of library vendors have chosen to use this technique.

You probably need to use platform specific APIs directly, I believe that they have much higher limits on the number of open files. MSDN documentation for the Win32API's CreateFile function doesn't seem to mention that there is an upper limit at all.

files open in C

Reply #5 –
Quote
Hi,

I tried out this code segment is VC 6


  for(i=0; i<1000; i++){
     stream = fopen("filename0xxx.txt", "w");
  }


where "filename0xxx.txt", xxx is different for different value of i..

My program will crash! It seemed that there is a upper limit to the number of files that a program can open this way.. Is there any way to increased the number of files that can be opened .. Maybe 10 000 files ? 

Most operating systems allocate a table of file descriptors of fixed size to each process.  There are ways to adjust the size of this table, which vary from one system to the next.  In general, you shouldn't assume that you're going to be able to keep huge numbers of open files.

I'm having a hard time coming up with a good reason to do this though.  The resources associated with each open file would probably be exhausted pretty quickly with so many open files (memory, etc).

files open in C

Reply #6 –
So, the limit is also imposed by the operating system ?

For serious R&D purposes, there is a need to keep thousands of files open ! In AAC for instance, the Main Predictors has 672 individual predictors and each predictor has many parameters which behaviors & performance needed to be tracked and analysed for every input audio frame.. You can end up with many files for a very complex software engineering project !

Of course, I could keep everything inside just one file and then design somekind of software tools to extract / interpret the individual data in this one single gigantic file ! But I thought that it is much more systematic to have all these data to be arranged in many different individual files..

files open in C

Reply #7 –
Quote
CreateFile[/URL] function doesn't seem to mention that there is an upper limit at all.


I think I tried out a simple program with CreateFile and it also had a upper limit just as with fopen..

files open in C

Reply #8 –
Quote
I think I tried out a simple program with CreateFile and it also had a upper limit just as with fopen..

Okay, so the operating system indeed does impose a limit. Pretty hard to circumvent that one

Quote
Of course, I could keep everything inside just one file and then design somekind of software tools to extract / interpret the individual data in this one single gigantic file !


This seems pretty logical to me. Design or use some existing file format from which you can extract individual data streams, maybe length-prefixed arrays of bytes or something. I don't think that should be exceedingly difficult.

However, if you could manage to cut down the number of simultaneously open files to a certain number, say 672, you could be able to work with individual files.

files open in C

Reply #9 –
Quote
Quote
CreateFile[/URL] function doesn't seem to mention that there is an upper limit at all.


I think I tried out a simple program with CreateFile and it also had a upper limit just as with fopen..

It depends on the OS too, Windows 95 (and the variants) have far less handles than those built on the NT core, such as XP and 2000. Same goes for GDI handles.

files open in C

Reply #10 –
I can confirm there is no such limit in C# and .NET (I've created 10000 files at time).
I know that this thread is about C, but what about trying Managed C++ (or C# of course  ) and .NET Framework?
mikeson

files open in C

Reply #11 –
Quote
I can confirm there is no such limit in C# and .NET (I've created 10000 files at time).
I know that this thread is about C, but what about trying Managed C++ (or C# of course  ) and .NET Framework?


Have you writtened a small program to test this problem in C# including writing some random data into each of the 10 000 files ?

files open in C

Reply #12 –
Quote
Have you writtened a small program to test this problem in C# including writing some random data into each of the 10 000 files ?

Yes, for example with this code:

Code: [Select]
namespace ProjectSoft.User
{
    using System;
    using System.IO;
    
    public class Runner
    {
  public static void Main()
  {
      StreamWriter[] myWriters = new StreamWriter[10000];

      for (int i = 0; i < myWriters.Length; i++)
      {
    myWriters[i] = new StreamWriter(String.Format("{0:0000}.log",i));
    myWriters[i].Write(String.Format("{0:0000}.log",i));
    myWriters[i].Flush();
      }
  }
    }
}


[EDIT] Some cleaning, replaced .Close() with .Flush() and removed statement using(myWriters = new...), because caused disposal of object (calling method Dispose()) in the end of scope and that was not intended (=close file)[/EDIT]
mikeson

files open in C

Reply #13 –
hmmmm.. funny that it is not an operating system problem.. I have not tried out C# yet so I can't be sure.. but you close the stream before opening a new stream..

What we are talking here is keeping 10 000 streams open at all time !

files open in C

Reply #14 –
You can replace myWriters.Close(); with myWriters.Flush(); and keep stream open and it will still work.
mikeson

files open in C

Reply #15 –
I tried with this code and it works fine:

Code: [Select]
      StreamWriter[] myWriters = new StreamWriter[10000];

      for (int i = 0; i < myWriters.Length; i++) {
    myWriters[i] = new StreamWriter(String.Format(@"G:\test\{0:0000}.log",i));
      }

      for (int i = 0; i < myWriters.Length; i++) {
    myWriters[i].Write(String.Format("Writing to log {0:0000}",i));
      }

      for (int i = 0; i < myWriters.Length; i++) {
    myWriters[i].Close();
      }

I'd like to try it with CreateFile in C++ as well.

files open in C

Reply #16 –
Here's the code I used to test CreateFile, I was able to open 10,000 files (didn't try any more than that):

Code: [Select]
#include <stdio.h>
#include <windows.h>

int main()
{
    const int numFiles = 10000;
    HANDLE hFiles[numFiles];
    char buff[256];
    int i;

    for (i = 0; i < numFiles; i++) {
  sprintf(buff, "G:\\test\\%05d.txt", i);

  hFiles[i] = CreateFile(buff, GENERIC_WRITE, 0, NULL, CREATE_NEW, FILE_ATTRIBUTE_NORMAL, 0);
  if (hFiles[i] == INVALID_HANDLE_VALUE) {
      printf("Unable to create file %d\n", i);
  }
    }

    for (i = 0; i < numFiles; i++) {
  if (hFiles[i] != INVALID_HANDLE_VALUE) {
      DWORD bytesWritten;
      sprintf(buff, "Writing to file %d", i);

      if (!WriteFile(hFiles[i], buff, strlen(buff), &bytesWritten, NULL)) {
    printf("Unable to write to file %d\n", i);
      }
  }
    }

    for (i = 0; i < numFiles; i++) {
  if (hFiles[i] != INVALID_HANDLE_VALUE) {
      CloseHandle(hFiles[i]);
  }
    }

    return 0;
}

I'm using Windows 2000.

files open in C

Reply #17 –
Thanks..  I think you solved my problem !! I must have used OpenFile instead of CreateFile sometime ago !

files open in C

Reply #18 –
Quote
Thanks..  I think you solved my problem !! I must have used OpenFile instead of CreateFile sometime ago !

Yes, this could very much be the case. OpenFile was deprecated in favour of CreateFile for Win32.

From MSDN:
Quote
Note This function is provided only for compatibility with 16-bit versions of Windows. New applications should use the CreateFile function.

Apparently OpenFile has some serious limitations compared to CreateFile.