https://111qqz.com/en/tags/hmm/Recent content in HMM on 111qqz's blogHugo -- gohugo.ioenhust.111qqz@gmail.com (111qqz)hust.111qqz@gmail.com (111qqz)© 2011-2026 111qqzSat, 10 Oct 2026 14:30:00 +0800https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-10-markov-property-knowledge-associations/Sat, 10 Oct 2026 14:30:00 +0800hust.111qqz@gmail.com (111qqz)https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-10-markov-property-knowledge-associations/<h2 class="relative group">起因 <div id="起因" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e8%b5%b7%e5%9b%a0" aria-label="Anchor">#</a> </span> </h2> <p>继续跟着 MIT 6.S191 Lecture 5 学强化学习。<a href="https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-09-rl-return-to-q-function/" >上一篇</a>理解了 Return、State Value Function \(V^\pi(s)\) 和 Action Value Function \(Q^\pi(s,a)\)。</p> <p>接下来要学的是 Bellman Equation。</p> <p>我之前隐约知道它能把 \(V^\pi(s)\) 写成递归形式——当前状态的长期价值，等于下一步的期望 Reward 加上折扣后下一状态的期望长期价值。但刚准备细看推导时，发现它有一个前提条件：</p>