1. **Problem Statement:** We need to develop and train a simple neural network using the backpropagation algorithm with a given dataset. 2. **Setup:** Let's consider a simple neural network with: - 2 input neurons - 2 neurons in one hidden layer - 1 output neuron 3. **Data:** We will use the XOR problem as the dataset: | Input 1 | Input 2 | Output | |---------|---------|--------| | 0 | 0 | 0 | | 0 | 1 | 1 | | 1 | 0 | 1 | | 1 | 1 | 0 | 4. **Algorithm:** Backpropagation uses gradient descent to minimize the error between predicted and actual outputs. 5. **Formulas:** - Activation function (sigmoid): $$\sigma(x) = \frac{1}{1 + e^{-x}}$$ - Derivative of sigmoid: $$\sigma'(x) = \sigma(x)(1 - \sigma(x))$$ - Error: $$E = \frac{1}{2} (target - output)^2$$ 6. **Steps:** - Initialize weights randomly. - For each training example: - Forward pass: compute outputs of hidden and output layers. - Compute error at output. - Backward pass: compute gradients and update weights. 7. **Example Calculation for one training input (0,1) with target 1:** - Assume initial weights and biases: - Input to hidden weights: $$w_{1}=0.15, w_{2}=0.20, w_{3}=0.25, w_{4}=0.30$$ - Hidden to output weights: $$w_{5}=0.40, w_{6}=0.45$$ - Biases: $$b_{hidden}=0.35, b_{output}=0.60$$ - Forward pass: - Hidden layer inputs: $$h_1 = 0*0.15 + 1*0.20 + 0.35 = 0.55$$ $$h_2 = 0*0.25 + 1*0.30 + 0.35 = 0.65$$ - Hidden layer outputs: $$o_{h1} = \sigma(0.55) \approx 0.634$$ $$o_{h2} = \sigma(0.65) \approx 0.657$$ - Output layer input: $$o = 0.634*0.40 + 0.657*0.45 + 0.60 = 1.197$$ - Output: $$o_{out} = \sigma(1.197) \approx 0.768$$ - Error: $$E = \frac{1}{2} (1 - 0.768)^2 = 0.0267$$ - Backward pass: - Output layer delta: $$\delta_{out} = (target - o_{out}) \times \sigma'(1.197) = (1 - 0.768) \times 0.768 \times (1 - 0.768) \approx 0.043$$ - Hidden layer deltas: $$\delta_{h1} = \delta_{out} \times w_5 \times \sigma'(0.55) = 0.043 \times 0.40 \times 0.232 = 0.004$$ $$\delta_{h2} = \delta_{out} \times w_6 \times \sigma'(0.65) = 0.043 \times 0.45 \times 0.224 = 0.004$$ - Update weights (learning rate $\eta=0.5$): - $$w_5 = w_5 + \eta \times \delta_{out} \times o_{h1} = 0.40 + 0.5 \times 0.043 \times 0.634 = 0.414$$ - $$w_6 = w_6 + \eta \times \delta_{out} \times o_{h2} = 0.45 + 0.5 \times 0.043 \times 0.657 = 0.464$$ - $$w_2 = w_2 + \eta \times \delta_{h1} \times input_2 = 0.20 + 0.5 \times 0.004 \times 1 = 0.202$$ - $$w_4 = w_4 + \eta \times \delta_{h2} \times input_2 = 0.30 + 0.5 \times 0.004 \times 1 = 0.302$$ 8. **Summary:** By repeating these steps for all training examples and multiple epochs, the network learns to approximate the XOR function. This is a simplified example illustrating the backpropagation algorithm in a neural network.