Unable to allocate array with shape and data type
This is likely due to your system's overcommit handling mode.
In the default mode, 0
,
Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. The root is allowed to allocate slightly more memory in this mode. This is the default.
The exact heuristic used is not well explained here, but this is discussed more on Linux over commit heuristic and on this page.
You can check your current overcommit mode by running
$ cat /proc/sys/vm/overcommit_memory
0
In this case, you're allocating
>>> 156816 * 36 * 53806 / 1024.0**3
282.8939827680588
~282 GB and the kernel is saying well obviously there's no way I'm going to be able to commit that many physical pages to this, and it refuses the allocation.
If (as root) you run:
$ echo 1 > /proc/sys/vm/overcommit_memory
This will enable the "always overcommit" mode, and you'll find that indeed the system will allow you to make the allocation no matter how large it is (within 64-bit memory addressing at least).
I tested this myself on a machine with 32 GB of RAM. With overcommit mode 0
I also got a MemoryError
, but after changing it back to 1
it works:
>>> import numpy as np
>>> a = np.zeros((156816, 36, 53806), dtype='uint8')
>>> a.nbytes
303755101056
You can then go ahead and write to any location within the array, and the system will only allocate physical pages when you explicitly write to that page. So you can use this, with care, for sparse arrays.
MemoryError: Unable to allocate 30.4 GiB for an array with shape (725000, 277, 76) and data type float64
Taking into account the dimensions of the dataset you provided (725000
x 277
x 76
) and its data type (float64 - 8
bytes), it seems that you need (at minimum) around 114
GB to have the dataset loaded/stored in RAM.
A solution to overcome this limitation is to: 1) read a certain amount of the dataset (e.g. a chunk of 1 GB at the time) through a hyperslab selection and load/store it in memory, 2) process it, and 3) repeat the process (i.e. go to step 1) until the dataset is completely processed. This way, you will not run out of RAM memory.
MemoryError: Unable to allocate MiB for an array with shape and data type, when using anymodel.fit() in sklearn
Upgrading python-64 bit seems to have solved all the "Memory Error" problem.
MemoryError: unable to allocate array with shape and data type float32 while using word2vec in python
Ideally, you should paste the text of your error into your question, rather than a screenshot. However, I see the two key lines:
<TIMESTAMP> : INFO : estimated required memory for 2372206 words and 400 dimensions: 8777162200 bytes
...
MemoryError: unable to allocate array with shape (2372206, 400) and data type float32
After making one pass over your corpus, the model has learned how many unique words will survive, which reports how large of a model must be allocated: one taking about 8777162200 bytes
(about 8.8GB). But, when trying to allocate the required vector array, you're getting a MemoryError
, which indicates not enough computer addressable-memory (RAM) is available.
You can either:
- run where there's more memory, perhaps by adding RAM to your existing system; or
- reduce the amount of memory required, chiefly by reducing either the number of unique word-vectors you'd like to train, or their dimensional size.
You could reduce the number of words by increasing the default min_count=5
parameter to something like min_count=10
or min_count=20
or min_count=50
. (You probably don't need over 2 million word-vectors – many interesting results are possible with just a vocabulary of a few tens-of-thousands of words.)
You could also set a max_final_vocab
value, to specify an exact number of unique words to keep. For example, max_final_vocab=500000
would keep just the 500000 most-frequent words, ignoring the rest.
Reducing the size
will also save memory. A setting of size=300
is popular for word-vectors, and would reduce the memory requirements by a quarter.
Together, using size=300, max_final_vocab=500000
should trim the required memory to under 2GB.
Unable to allocate memory with array shape to create reinforcement learning model
It looks like you simply don't have enough RAM to allocate 229 GiB for an array of that size---which is incredibly large---very few computers could.
Have you tried batching your idea into batches of either 64, 128, 256, etc.? That is a very common way to decrease the memory load, and you can experiment with different values to see what computation you can handle. Tensorflow has a great deal of built-in methods that can help you here. One place to look would be the batch method here.
Related Topics
Pandas Filtering for Multiple Substrings in Series
Programme to Print Mulitples of 5 in a Range Specified by User
Move Files Between Two Aws S3 Buckets Using Boto3
How to Extract Hours and Minutes from a Datetime.Datetime Object
How to Get the Response Json Data from Network Call in Xhr Using Python Selenium Web Driver Chorme
Vscode Import Error for Python Module
Print() Prints Only Every Second Input
Find the Index of a Value in a 2D Array
Convert Images from [-1; 1] to [0; 255]
Replacing All Negative Values in Certain Columns by Another Value in Pandas
How to Convert a 1 Channel Image into a 3 Channel With Opencv2
How to Download the Latest File of an S3 Bucket Using Boto3
How to Add Parenthesis Around a Substring in a String
Possible to Loop Through Excel Files With Differently Named Sheets, and Import into a List
Using Regex to Get the Value Between Two Characters (Python 3)