valueerror assignment destination is read only joblib

ValueError: assignment destination is read-only [Solved]

Last updated: Apr 11, 2024 Reading time · 2 min

# ValueError: assignment destination is read-only [Solved]

The NumPy "ValueError: assignment destination is read-only" occurs when you try to assign a value to a read-only array.

To solve the error, create a copy of the read-only array and modify the copy.

You can use the flags attribute to check if the array is WRITABLE .

Running the code sample produces the following output:

In your case, WRITEABLE will likely be set to False .

In older NumPy versions, you used to be able to set the flag to true by calling the setflags() method.

However, setting the WRITEABLE flag to True ( 1 ) will likely fail if the OWNDATA flag is set to False .

You will likely get the following error:

• "ValueError: cannot set WRITEABLE flag to True of this array"

To solve the error, create a copy of the array when converting it from a Pillow Image to a NumPy array.

Passing the Pillow Image to the numpy.array() method creates a copy of the array.

You can also explicitly call the copy() method.

You can also call the copy() method on the Pillow Image and modify the copy.

The image copy isn't read-only and allows assignment.

You can change the img_copy variable without getting the "assignment destination is read-only" error.

You can also use the numpy.copy() method.

The numpy.copy() method returns an array copy of the given object.

The only argument we passed to the method is the image.

You can safely modify the img_copy variable without running into issues.

You most likely don't want to make changes to the original image.

Creating a copy and modifying the copy should be your preferred approach.

If you got the error when using the np.asarray() method, try changing it to np.array() .

Change the following:

To the following:

As long as the WRITEABLE flag is set to True , you will be able to modify the array.

# Additional Resources

You can learn more about the related topics by checking out the following tutorials:

• TypeError: Object of type ndarray is not JSON serializable
• ValueError: numpy.ndarray size changed, may indicate binary incompatibility
• NumPy RuntimeWarning: divide by zero encountered in log10
• ValueError: x and y must have same first dimension, but have shapes

Borislav Hadzhiev

Web Developer

Copyright © 2024 Borislav Hadzhiev

joblib / joblib

Computing with Python functions.

Home Page: http://joblib.readthedocs.org

Geek Repo: Geek Repo

Github PK Tool: Github PK Tool

When using the Catboost model to call sklearn's permanence importance to calculate the contribution rate, the joblib library prompts ValueError: assignment destination is read-only

qq492947833 opened this issue 2 years ago · comments

As an example, when I use the Catboost model to call sklearn's permanence importance to calculate the contribution rate, the joblib library prompts ValueError: assignment destination is read-only. This cannot be solved by modifying max nbytes, even if I increase it to 99999M. This problem only occurs when I use catboost, not when I use other GBDT models.I want to know why. If you can answer, I would be very grateful!

We had some problems with the new version of scipy dependency. Maybe it could help you.

We forced to install the previous version scipy==1.9.1 instead of last scipy==1.9.2 installed by default as scikit-learn dependecy.

Could you please provide the Python source code for a minimal reproduction case?

https://scikit-learn.org/dev/developers/minimal_reproducer.html#crafting-a-minimal-reproducer-for-scikit-learn

We had some problems with the new version of scipy dependency. Maybe it could help you. We forced to install the previous version scipy==1.9.1 instead of last scipy==1.9.2 installed by default as scikit-learn dependecy.

Thank you very much,it is worked.

Serialization #

Since Ray processes do not share memory space, data transferred between workers and nodes will need to serialized and deserialized . Ray uses the Plasma object store to efficiently transfer objects across different processes and different nodes. Numpy arrays in the object store are shared between workers on the same node (zero-copy deserialization).

Ray has decided to use a customized Pickle protocol version 5 backport to replace the original PyArrow serializer. This gets rid of several previous limitations (e.g. cannot serialize recursive objects).

Ray is currently compatible with Pickle protocol version 5, while Ray supports serialization of a wider range of objects (e.g. lambda & nested functions, dynamic classes) with the help of cloudpickle.

Plasma Object Store #

Plasma is an in-memory object store. It has been originally developed as part of Apache Arrow. Prior to Ray’s version 1.0.0 release, Ray forked Arrow’s Plasma code into Ray’s code base in order to disentangle and continue development with respect to Ray’s architecture and performance needs.

Plasma is used to efficiently transfer objects across different processes and different nodes. All objects in Plasma object store are immutable and held in shared memory. This is so that they can be accessed efficiently by many workers on the same node.

Each node has its own object store. When data is put into the object store, it does not get automatically broadcasted to other nodes. Data remains local to the writer until requested by another task or actor on another node.

Serializing ObjectRefs #

Explicitly serializing ObjectRefs using ray.cloudpickle should be used as a last resort. Passing ObjectRefs through Ray task arguments and return values is the recommended approach.

Ray ObjectRefs can be serialized using ray.cloudpickle . The ObjectRef can then be deserialized and accessed with ray.get() . Note that ray.cloudpickle must be used; other pickle tools are not guaranteed to work. Additionally, the process that deserializes the ObjectRef must be part of the same Ray cluster that serialized it.

When serialized, the ObjectRef ’s value will remain pinned in Ray’s shared memory object store. The object must be explicitly freed by calling ray._private.internal_api.free(obj_ref) .

ray._private.internal_api.free(obj_ref) is a private API and may be changed in future Ray versions.

This code example demonstrates how to serialize an ObjectRef , store it in external storage, deserialize and use it, and lastly free its object.

Numpy Arrays #

Ray optimizes for numpy arrays by using Pickle protocol 5 with out-of-band data. The numpy array is stored as a read-only object, and all Ray workers on the same node can read the numpy array in the object store without copying (zero-copy reads). Each numpy array object in the worker process holds a pointer to the relevant array held in shared memory. Any writes to the read-only object will require the user to first copy it into the local process memory.

You can often avoid serialization issues by using only native types (e.g., numpy arrays or lists/dicts of numpy arrays and other primitive types), or by using Actors hold objects that cannot be serialized.

Fixing “assignment destination is read-only” #

Because Ray puts numpy arrays in the object store, when deserialized as arguments in remote functions they will become read-only. For example, the following code snippet will crash:

To avoid this issue, you can manually copy the array at the destination if you need to mutate it ( arr = arr.copy() ). Note that this is effectively like disabling the zero-copy deserialization feature provided by Ray.

Serialization notes #

Ray is currently using Pickle protocol version 5. The default pickle protocol used by most python distributions is protocol 3. Protocol 4 & 5 are more efficient than protocol 3 for larger objects.

For non-native objects, Ray will always keep a single copy even it is referred multiple times in an object:

Whenever possible, use numpy arrays or Python collections of numpy arrays for maximum performance.

Lock objects are mostly unserializable, because copying a lock is meaningless and could cause serious concurrency problems. You may have to come up with a workaround if your object contains a lock.

Customized Serialization #

Sometimes you may want to customize your serialization process because the default serializer used by Ray (pickle5 + cloudpickle) does not work for you (fail to serialize some objects, too slow for certain objects, etc.).

There are at least 3 ways to define your custom serialization process:

If you want to customize the serialization of a type of objects, and you have access to the code, you can define __reduce__ function inside the corresponding class. This is commonly done by most Python libraries. Example code:

<sqlite3.Connection object at ...> <sqlite3.Connection object at ...>

If you want to customize the serialization of a type of objects, but you cannot access or modify the corresponding class, you can register the class with the serializer you use:

NOTE: Serializers are managed locally for each Ray worker. So for every Ray worker, if you want to use the serializer, you need to register the serializer. Deregister a serializer also only applies locally.

If you register a new serializer for a class, the new serializer would replace the old serializer immediately in the worker. This API is also idempotent, there are no side effects caused by re-registering the same serializer.

We also provide you an example, if you want to customize the serialization of a specific object:

Troubleshooting #

Use ray.util.inspect_serializability to identify tricky pickling issues. This function can be used to trace a potential non-serializable object within any Python object – whether it be a function, class, or object instance.

Below, we demonstrate this behavior on a function with a non-serializable object (threading lock):

The resulting output is:

For even more detailed information, set environmental variable RAY_PICKLE_VERBOSE_DEBUG='2' before importing Ray. This enables serialization with python-based backend instead of C-Pickle, so you can debug into python code at the middle of serialization. However, this would make serialization much slower.

Known Issues #

Users could experience memory leak when using certain python3.8 & 3.9 versions. This is due to a bug in python’s pickle module .

This issue has been solved for Python 3.8.2rc1, Python 3.9.0 alpha 4 or late versions.

Valueerror assignment destination is read-only

One of the errors that developers often come across is the ValueError: assignment destination is read-only .

This error typically occurs when you try to modify a read-only object or variable.

What does the ValueError: assignment destination is read-only error mean?

This error occurs when you try to modify a read-only object or variable in your code. The error message indicates that the assignment destination is restricted to being read-only.

How the Error Occurs?

These are the following examples of how the error occurs.

Example 1: Modifying an Immutable Tuple

• The code creates a tuple called “my_tuple” with the values 1, 2, and 3.
• Tuples are immutable, meaning their elements cannot be changed once they are created.
• The line “my_tuple[0] = 10” attempts to modify the value at index 0 of the tuple to 10, which is not allowed and will result in a valueerror.

Example 2: Attempting to Modify a String

The code attempts to change the first character of a string but produces a valueerror because strings cannot be directly modified.

Example 3: Assigning to a Constant

The code example attempts to assign a new value (3.14) to the constant math.pi in Python.

However, in Python, constants are typically immutable and cannot be modified once defined.

Therefore, this code will likely raise a valueerror .

Example 4: Altering an Immutable Data Structure

This code example demonstrates how to modify a value in an immutable data structure in Python.

It starts with a dictionary called “ my_dict ” that has a key-value pair of ‘ key ‘ and ‘ value ‘.

The code then changes the value associated with the ‘key’ to ‘ new_value ‘ by assigning it directly using indexing.

Solutions for ValueError: assignment destination is read-only

Here are some solutions to solve the ValueError: assignment destination is read-only:

Solution 1: Use Mutable Data Structures

Instead of using immutable data structures like tuples or strings, switch to mutable ones such as lists or dictionaries.

Mutable objects allow modifications, eliminating the read-only error.

For example:

Solution 2: Reassign Variables

If you encounter the valueerror while attempting to modify a variable, try reassigning it with a new value.

This method can help you resolve the read-only restriction.

Solution 3: Check Documentation and Restrictions

In some cases, certain objects or variables are intentionally designed to be read-only.

It’s important to consult the documentation or source code to understand any restrictions required.

Make sure that the object you’re attempting to modify is meant to be changed.

Solution 4: Use a Copy or Clone

If the object you’re working with is meant to be read-only, consider creating a copy or clone of it.

By doing this, you can modify the duplicate without affecting the original read-only object.

Solution 5: Identify Context-Specific Solutions

The ValueError: assignment destination is read-only error that can be context-specific.

Analyze the specific context where the error occurs and seek solutions to that scenario.

Understanding the possible cause will aid in finding a proper resolution.

Solution 6: Seek Help from the Community

If all the solutions are not working, don’t hesitate to reach out to the programming community for assistance.

Online forums, developer communities, and platforms like Stack Overflow can provide valuable insights and guidance from experienced programmers.

Frequently Asked Questions

To resolve this valueerror, you can apply different solutions such as using mutable data structures, reassigning variables, checking documentation and restrictions, using copies or clones, identifying context-specific solutions, or seeking help from the programming community.

The error arises because the variable you are attempting to modify is marked as read-only. It could be intentionally designed this way or due to a restriction imposed by the programming language or framework you are using.

In some cases, you can convert a read-only object into a writable one by using techniques like creating a copy or clone of the object. However, this depends on the specific context and the restrictions imposed on the program.

Yes, there are similar errors that you may encounter in different programming languages. For example, in JavaScript, you might encounter the error TypeError: Assignment to a constant variable when trying to modify a constant.

The ValueError: assignment destination is read-only error can be encountered when attempting to modify read-only objects or variables in your code.

By following the solutions provided in this article, such as using mutable data structures, reassigning variables, checking restrictions, making copies or clones, considering context-specific solutions, and seeking community help, you can effectively resolve this error.

Additional Resources

• Valueerror: could not convert tuple of form to variable
• Valueerror unconverted data remains
• Valueerror index contains duplicate entries cannot reshape

Leave a Comment Cancel reply

You must be logged in to post a comment.

Determine how a param is being set as readonly

I have a class similar to the example below

I’m using this basic example to test the usage pattern where I store a numpy.ndarray once but keep both a numpy recarray and pandas Dataframe view of the same data to allow for different access patterns a viewing with panel without duplicating the large array in memory.

this example works as you can see from the example below

However I have this same basic pattern in a much larger class but when I try to assign a value to the main numpy array I get a ValueError: assignment destination is read-only . I have not declared any of the paramerters in param.Parameterized class as readonly or constant but continue to get this simple ValueError.

Is there a way to track down were the readonly designation is being applied via some other more detailed stack trace? I’m beyond frustrated with trying ot figure out why the main array is being assigned as readonly when I have made no such designation in my code.

I have found the source of my error. It appears this is an issue associated with reading numpy arrays from a buffer not with the parameters themselves.

Hopefully this helps someone avoid a couple of days of frustrating searching in the future.

If you read a numpy array in from a buffer and would like to manipulate the values across views add a .copy() to the end of the initial read so that the array is not set as read-only.

Just out of curiosity, did the error message ( ValueError: assignment ... ) have anything to do with Param itself?

No. I have been really impressed with param so I’m rewriting my old code into param.Parameterized classes to better document the purpose of the code and to take advantage of the visualization capabilities of panel. One of the features I was using in Jupyter as I get better at parm and panel was the with param.exceptions_summarized(): context manager.

I mistakenly thought this context manager would only summarize the param related exceptions since the code was previously working as non-param classes. The ValueError was the only thing being dumped out so I assumed it was a param related error. Once I removed the context manager and explored the full trace I found the numpy related issue. This more a gap in my understanding than a real issue with any one library.

Thanks for all the help. My 2022 resolution is to get all my code into param and panel related classes so I’m sure I will be pestering everyone with issues that may end up being more me than the libraries themselves. Merry Christmas!

I believe exceptions_summarized was added to param purely to write the documentation, it’s useful to only print the error message of an instead long and boring exception traceback, exception that would stop the notebook execution. But in your code you should definitely not use it since it will skip exceptions. This context manager would be better off in a separate module dedicated to the docs.

System error: assignment destination is read-only

• High: It blocks me to complete my task.

I would like to ask about an issue that I encountered when I try to distribute my work on multiple cpu nodes using ray.

My input file is a simulation file consisting of multiple time frames, so I would like to distribute the calculation of one frame to one task. It works fine when I just used pool from the multiprocessing python library, where only one node (128 tasks in total) can be used. Since I have more than 2,000 time frames, I would like to use multiple nodes in this calculation, and the multiprocessing python library isn’t the best choice.

I created my code using this template: ray/simple-trainer.py at master · ray-project/ray · GitHub . Here’s a brief summary of my code:

import socket import sys import time import ray

@ray.remote def hydration_water_calculation2(t, u): # in one frame return xyz

ray.init(address=os.environ[“ip_head”])

print(“Nodes in the Ray cluster:”) print(ray.nodes())

for i in frame_values: ip_addresses = ray.get([hydration_water_calculation2.remote(i, u0) for _ in range(1)]) print(Counter(ip_addresses))

But I got the following error: Traceback (most recent call last): File “…/…/hydration_whole_global2_ray.py”, line 269, in ip_addresses = ray.get([hydration_water_calculation2.remote(i, u0) for _ in range(1)]) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/_private/client_mode_hook.py”, line 105, in wrapper return func(*args, **kwargs) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/worker.py”, line 1809, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError: ray::hydration_water_calculation2() (pid=27283, ip=10.8.9.236) At least one of the input arguments for this task could not be computed: ray.exceptions.RaySystemError: System error: assignment destination is read-only traceback: Traceback (most recent call last): File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 332, in deserialize_objects obj = self._deserialize_object(data, metadata, object_ref) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 235, in _deserialize_object return self._deserialize_msgpack_data(data, metadata_fields) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 190, in _deserialize_msgpack_data python_objects = self._deserialize_pickle5_data(pickle5_data) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 178, in _deserialize_pickle5_data obj = pickle.loads(in_band, buffers=buffers) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 2106, in setstate self[self.ts.frame] File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 1610, in getitem return self._read_frame_with_aux(frame) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 1642, in _read_frame_with_aux ts = self._read_frame(frame) # pylint: disable=assignment-from-no-return File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XDR.py”, line 255, in _read_frame timestep = self._read_next_timestep() File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XDR.py”, line 273, in _read_next_timestep self._frame_to_ts(frame, ts) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XTC.py”, line 144, in _frame_to_ts ts.dimensions = triclinic_box(*frame.box) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 810, in dimensions self._unitcell[:] = box ValueError: assignment destination is read-only (hydration_water_calculation2 pid=27283) 2022-05-01 22:53:55,714 ERROR serialization.py:334 – assignment destination is read-only (hydration_water_calculation2 pid=27283) Traceback (most recent call last): (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 332, in deserialize_objects (hydration_water_calculation2 pid=27283) obj = self._deserialize_object(data, metadata, object_ref) (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 235, in _deserialize_object (hydration_water_calculation2 pid=27283) return self._deserialize_msgpack_data(data, metadata_fields) (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 190, in _deserialize_msgpack_data (hydration_water_calculation2 pid=27283) python_objects = self._deserialize_pickle5_data(pickle5_data) (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 178, in _deserialize_pickle5_data (hydration_water_calculation2 pid=27283) obj = pickle.loads(in_band, buffers=buffers) (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 2106, in setstate (hydration_water_calculation2 pid=27283) self[self.ts.frame] (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 1610, in getitem (hydration_water_calculation2 pid=27283) return self._read_frame_with_aux(frame) (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 1642, in _read_frame_with_aux (hydration_water_calculation2 pid=27283) ts = self._read_frame(frame) # pylint: disable=assignment-from-no-return (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XDR.py”, line 255, in _read_frame (hydration_water_calculation2 pid=27283) timestep = self._read_next_timestep() (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XDR.py”, line 273, in _read_next_timestep (hydration_water_calculation2 pid=27283) self._frame_to_ts(frame, ts) (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XTC.py”, line 144, in _frame_to_ts (hydration_water_calculation2 pid=27283) ts.dimensions = triclinic_box(*frame.box) (hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 810, in dimensions (hydration_water_calculation2 pid=27283) self._unitcell[:] = box (hydration_water_calculation2 pid=27283) ValueError: assignment destination is read-only

Could anyone help me diagnose the issue? I’m new to ray and still learning why I was getting the “assignment destination is read-only” error. Many thanks in advance!

Hey @Chengeng-Yang , the read-only errors are happening because Ray stores arguments in the shared memory object store. This allows arguments to be shared with process very efficiently with zero memory copies, but has a side-effect of rendering numpy arrays immutable.

In this case, it seems that during setstate for your program an assignment is made that will update an existing array. Is it possible to modify the code around there to make a copy of the array prior to calling self._unitcell[:] = box ? I.e., self._unitcell = self._unitcell.copy(); self._unitcell[:] = box . That should fix the deserialization problem.

Reference stack line: File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 2106, in setstate

Hi @ericl , many thanks for your help! Your suggestion DID work. The deserialization issue has been fixed after I made a copy of self._unitcell.

Thanks again and have a wonderful day :))

Related Topics

scikit-learn: ValueError: assignment destination is read-only, when paralleling with n_jobs > 1

When I run SparseCoder with n_jobs > 1, there is a chance to raise exception ValueError: assignment destination is read-only . The code is shown as follow:

The bigger data_dims is, the higher chance get. When data_dims is small (lower than 2000, I verified), everything works fine. Once data_dims is bigger than 2000, there is a chance to get the exception. When data_dims is bigger than 5000, it is 100% raised.

My version infor:

OS: OS X 10.11.1 python: Python 2.7.10 |Anaconda 2.2.0 numpy: 1.10.1 sklearn: 0.17

The full error information is shown as follow

About this issue

• Original URL
• State: closed
• Created 8 years ago
• Reactions: 4
• Comments: 23 (10 by maintainers)

Commits related to this issue

• FIX SparseCoder with readonly parallel mmap Fixes #5956. — committed to ogrisel/scikit-learn by ogrisel  6 years ago

Most upvoted comments

@coldfog @garciaev I don’t know if it is still relevant for you, but I ran into the same problem using joblib without scikit-learn.

The reason is the max_nbytes parameter within the Parallel invocation of the Joblib-library when you set n_jobs>1, which is 1M by default. The definition of this parameter is: “Threshold on the size of arrays passed to the workers that triggers automated memory mapping in temp_folder”. More details can be found here: https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html#

So, once the arrays pass the size of 1M, joblib will throw the error “ValueError: assignment destination is read-only”. In order to overcome this, the parameter has to be set higher, e.g. max_nbytes=‘50M’.

If you want a quick-fix, you can add max_nbytes=‘50M’ to the file “sklearn/decomposition/dict_learning.py” at line 297 in the Parallel class initiation to increase the allowed size of temporary files.

Just to complement @lvermue answer. I did what he suggested, but instead inside sklearn/decomposition/dict_learning.py the Parallel class that is instantiated doesn’t have the max_nbytes set. What worked for me was to increase the default max_nbytes inside sklearn/externals/joblib/parallel.py from “1M” to “10M” I think you can put more if needed.

@williamdjones I was not suggesting that it’s solved, but that it’s an issue that is reported at a different place, and having multiple issues related to the same problem makes keeping track of it harder.

I confirm that there is an error and it is floating in nature.

sklearn.decomposition.SparseCoder(D, transform_algorithm = ‘omp’, n_jobs=64).transform(X)

if X.shape[0] > 4000 it fails with ValueError: assignment destination is read-only If X.shape[0] <100 it is ok.

OS: Linux 3.10.0-327.13.1.el7.x86_64 python: Python 2.7.5 numpy: 1.10.1 sklearn: 0.17

Scikit-learn: ValueError: assignment destination is read-only, when paralleling with n_jobs > 1

When I run SparseCoder with n_jobs > 1, there is a chance to raise exception ValueError: assignment destination is read-only . The code is shown as follow:

The bigger data_dims is, the higher chance get. When data_dims is small (lower than 2000, I verified), everything works fine. Once data_dims is bigger than 2000, there is a chance to get the exception. When data_dims is bigger than 5000, it is 100% raised.

My version infor:

OS: OS X 10.11.1 python: Python 2.7.10 |Anaconda 2.2.0 numpy: 1.10.1 sklearn: 0.17

The full error information is shown as follow

Most helpful comment

@coldfog @garciaev I don't know if it is still relevant for you, but I ran into the same problem using joblib without scikit-learn.

The reason is the max_nbytes parameter within the Parallel invocation of the Joblib-library when you set n_jobs>1, which is 1M by default. The definition of this parameter is: "Threshold on the size of arrays passed to the workers that triggers automated memory mapping in temp_folder". More details can be found here: https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html#

So, once the arrays pass the size of 1M, joblib will throw the error "ValueError: assignment destination is read-only". In order to overcome this, the parameter has to be set higher, e.g. max_nbytes='50M'.

If you want a quick-fix, you can add max_nbytes='50M' to the file "sklearn/decomposition/dict_learning.py" at line 297 in the Parallel class initiation to increase the allowed size of temporary files.

All 23 comments

I am taking a look at this

Is it not related to #5481, which seems more generic?

It is, but SparseEncoder is not an estimator

Not that it matters but SparseCoder is an estimator:

I guess the error wasn't detected in #4807 as it is raised only when using algorithm='omp' . It should be raised when testing read only data on OrthogonalMatchingPursuit though.

Was there a resolution to this bug? I've run into something similar while doing n_jobs=-1 on RandomizedLogisticRegression, and didn't know whether I should open a new issue here. Here's the top of my stack:

Someone ran into the same exact problem on StackOverflow - ValueError: output array is read-only . Both provided solutions on SO are useless (the first one doesn't even bother solving the problem, and the second one is solving the problem by bypassing joblib completely).

@alichaudry I just commented on a similar issue here .

I confirm that there is an error and it is floating in nature.

sklearn.decomposition.SparseCoder(D, transform_algorithm = 'omp', n_jobs=64).transform(X)

if X.shape[0] > 4000 it fails with ValueError: assignment destination is read-only If X.shape[0] <100 it is ok.

OS: Linux 3.10.0-327.13.1.el7.x86_64 python: Python 2.7.5 numpy: 1.10.1 sklearn: 0.17

Hi there, I'm running into the same problem, using MiniBatchDictionaryLearning with jobs>1. I see a lot of referencing to other issues, but was there ever a solution to this? Sorry in advance if a solution was mentioned and I missed it.

OS: OSX python: 3.5 numpy: 1.10.1 sklearn: 0.17

The problem is in modifying arrays in-place. @lesteve close as duplicate of #5481?

currently I am still dealing with this issue and it is nearly a year since. this is still an open issue.

If you have a solution, please contribute it, @williamdjones

https://github.com/scikit-learn/scikit-learn/pull/4807 is probably the more advanced effort to address this.

@williamdjones I was not suggesting that it's solved, but that it's an issue that is reported at a different place, and having multiple issues related to the same problem makes keeping track of it harder.

Not sure where to report this, or if it's related, but I get the ValueError: output array is read-only when using n_jobs > 1 with RandomizedLasso and other functions.

@JGH1000 NOT A SOLUTION, but I would try using a random forest for feature selection instead since it is stable and has working joblib functionality.

Thanks @williamdjones , I used several different methods but found that RandomizedLasso works best for couple of particular datasets. In any case, it works but a bit slow. Not a deal breaker.

@JGH1000 No problem. If you don't mind, I'm curious about the dimensionality of the datasets for which RLasso was useful versus those for which it was not.

@williamdjones it was a small sample size (40-50), high-dimension (40,000-50,000) dataset. I would not say that other methods were bad, but RLasso provided results/ranking that were much more consistent with several univariate tests + domain knowledge. I guess this might not be the 'right' features but I had more trust in this method. Shame to hear it will be removed from scikit.

The problem still seems to exist on 24 core Ubuntu processor for RLasso with n_jobs = -1 and sklearn 0.19.1

Just to complement @lvermue answer. I did what he suggested, but instead inside sklearn/decomposition/dict_learning.py the Parallel class that is instantiated doesn't have the max_nbytes set. What worked for me was to increase the default max_nbytes inside sklearn/externals/joblib/parallel.py from "1M" to "10M" I think you can put more if needed.

And you can find max_nbytes at line 475 of file parallel.py

Related issues

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception "assignment destination is read-only" when reading from a read-only array #3758

khinsen commented Sep 17, 2013

• ❤️ 1 reaction

njsmith commented Sep 17, 2013

Sorry, something went wrong.

khinsen commented Sep 18, 2013

Njsmith commented sep 18, 2013.

charris commented Feb 23, 2014

No branches or pull requests