ValueError: Error when checking input: expected lstm_1_input to have shape (13100, 2) but got array with shape (2, 1)
I’m sure we’ve all seen this very annoying error before when using neural networks. I have personally spent weeks trying to debug my code in order to fix this error early in my Data Science Consultant career.
I want to make sure to address this error, how to understand the error and be prepared for when you see it again.
What does the error mean?
When you see the ValueError it usually means that your model (convolutional, lstm, etc.) ran into an input error. When you perform a neural network on datasets you will have to configure that dataset to comply with the input standards/expectations of the neural network.
If your message says ValueError: Error when checking input: expected conv1d_1_input to have shape (20000, 1) but got array with shape (1, 2) then that means that the input you gave did not match the input the neural network requires. If you are dealing with an LSTM or Conv1D then those neural networks expect an input shape that’s a 3D. Maybe your data is under the expectation of the neural network.
How to fix error?
Before you know how to fix it, you must learn the expected input shape that the neural network is looking for. A Conv1D expects an input shape as a 3D: [batch_size, time_steps, input_dimension], however, your dataset might be 2D: [batch_size, features]
Reshape data
The best way to fix the error is to expand your dataset dimension. You can do this by calling the .reshape method which belongs to pandas. You can also do np.expand_dim method to expand your dataset’s dimensions. The image above shows code of how I reshaped my own data to comply with a neural network that expects a 3D input shape. You can also see that I commented out some code, this is to show you that there are 2 ways of doing the same thing.
You can see that I put sample size, time steps, and input dimension as integer variables. This is to keep organized. Your sample size represents the number of rows. Sample size also is interchangeable with batch_size. Your time steps represent the number of features/independent variables that you’re using to train. The input dimension represents the extra dimension you put it, this usually remains at a value of 1.
For example, if you see an error that says: AttributeError: ‘DataFrame’ object has no attribute ‘reshape’ then this means that your train and test splits are not a Dataframe. This means you should use the np.reshape method instead. But if you did not get into an error then you are able to move forward to the neural network.
Neural Network input_shape parameter
Moving onto the neural, this is another area where you will receive an error. You have to make sure that you do an input_shape with the correct values, obviously or you wouldn't be reading this right. The parameter input_shape should be your from your new, updated, resized data.
If you’ve resized your data you have completed the first step. You might have chosen to stop reading because you believe that your error will now go away with the added dimension of your data. WRONG! you are now in a world full of hurt, breaking your keyboard and slamming your computer shut.
Maybe you’ve tried putting the input_shape parameter and batch_input_shape parameter with a value of (batch_size, time_steps, input_dimension) which will also give you that same annoying error or you’ve tried putting your 2D dimension in: X_train.shape Let’s fix this up:
To fix this you will have to go back to your neural network’s input_shape parameter and put the correct input shape of (time_step, input_dimension). To make it easier you can follow the code below which takes the new, reshaped data and call the .shape method and return index 1 and 2.
Now you should be able to run your code and watch as your model trains. Enjoy the wait!
BONUS ERROR
Wait! what if you are dealing with an LSTM model? After following the steps above did you just receive an error when running the neural network, saying: ValueError: Input 0 is incompatible with layer lstm_2: expected ndim=3, found ndim=2
This means that you might want to change one slight parameter, which is the boolean parameter return_sequences. If you have the value as False, change it to True and your code will run.
Outro
That’s all of the steps to make sure your project can go back to it’s regularly scheduled program. If you’d like to see the entire project then you can check out the GitHub repo.
Quick Plug
If you’re interested in Data Science Consultant Services then you can contact me by Email: zakirjefferson@gmail.com and LinkedIn. Free services are available on certain projects for companies!
