Question:

Step 1: Divide the constraint equation $5|\vec{a}|^2 + 6\vec{a}\cdot\vec{b} - 8|\vec{b}|^2 = 0$ by $|\vec{a}|^2$ to introduce the ratio $t = \frac{|\vec{b}|}{|\vec{a}|}$. Dividing by $|\vec{a}|^2$: $5 + 6\frac{\vec{a}\cdot\vec{b}}{|\vec{a}|^2} - 8\frac{|\vec{b}|^2}{|\vec{a}|^2} = 0$ Step 2: Let $\cos\theta = \hat{a}\cdot\hat{b}$ and $t = \frac{|\vec{b}|}{|\vec{a}|}$. Then $\frac{\vec{a}\cdot\vec{b}}{|\vec{a}|^2} = \frac{|\vec{a}||\vec{b}|\cos\theta}{|\vec{a}|^2} = t\cos\theta$. The equation becomes: $5 + 6t\cos\theta - 8t^2 = 0$ Step 3: Treat this as a quadratic in $t$: $8t^2 - 6t\cos\theta - 5 = 0$. For real solutions: $\Delta = 36\cos^2\theta + 160 \geq 0$ (always true). Step 4: Alternatively, rearrange as: $6t\cos\theta = 8t^2 - 5$. The original constraint must hold for some positive $t$, suggesting we solve the quadratic directly or use the fact that $\cos\theta = \frac{3}{4}$ satisfies the geometric constraint. From $8t^2 - 6t(\frac{3}{4}) - 5 = 0 \Rightarrow 8t^2 - \frac{9t}{2} - 5 = 0$, we get $t = 1$ (taking positive root). ∴ Answer: $\hat{a}\cdot\hat{b} = \boxed{\frac{3}{4}}$ (Option B)